[gRPC/cache] Received message larger than max

[graphicore]

The reason is a message size limitation on the gRPC side. We can work around this.
Also: it would be good to compress the messages that are on the wire and put into the cache.

One workaround could be to find the right gRPC setting to allow for bigger messages.
Another one would be to send big messages in smaller chunks, gRPC already has a streaming interface, so we can use that.

Maybe helpful:

• https://github.com/grpc/grpc/blob/master/include/grpc/impl/codegen/grpc_types.h#L148
• Allow specifying optional parameters for gRPC googleapis/gax-nodejs#107
• common-grpc: set grpc defaults and pass metadata googleapis/google-cloud-node#2007

2017-12-17T15:00:56.797286429Z DEBUG cache [PUT] with 1 payloads
2017-12-17T15:00:56.866007656Z DEBUG _dispatchFamilyJob: Caudex GitHub-GoogleFonts/pulls
2017-12-17T15:00:56.867518963Z INFO sendToQueue:  fontbakery-manifest-master-jobs
2017-12-17T15:00:56.957800492Z DEBUG cache [PUT] with 1 payloads
2017-12-17T15:01:04.675415508Z DEBUG _dispatchFamilyJob: GFS Neohellenic GitHub-GoogleFonts/pulls
2017-12-17T15:01:04.686995848Z INFO sendToQueue:  fontbakery-manifest-master-jobs 
 2017-12-17T15:01:04.701385066Z ERROR cache [PUT] on:error { Error: Received message larger than max (70491761 vs. 4194304)
2017-12-17T15:01:04.701419191Z     at ClientDuplexStream._emitStatusIfDone (/var/javascript/node_modules/grpc/src/client.js:255:19)
2017-12-17T15:01:04.701425618Z     at ClientDuplexStream._receiveStatus (/var/javascript/node_modules/grpc/src/client.js:233:8)
2017-12-17T15:01:04.701429641Z     at /var/javascript/node_modules/grpc/src/client.js:757:12 code: 8, metadata: Metadata { _internal_repr: {} } }
2017-12-17T15:01:04.701670484Z WARNING cache [PUT] on:status { code: 8,
2017-12-17T15:01:04.701701914Z   details: 'Received message larger than max (70491761 vs. 4194304)',
2017-12-17T15:01:04.701708448Z   metadata: Metadata { _internal_repr: {} } }
2017-12-17T15:01:04.701738673Z DEBUG cache [PUT] with 1 payloads
2017-12-17T15:01:05.624892819Z ERROR cache [PUT] on:error { Error: Received message larger than max (15221287 vs. 4194304)
2017-12-17T15:01:05.624931804Z     at ClientDuplexStream._emitStatusIfDone (/var/javascript/node_modules/grpc/src/client.js:255:19)
2017-12-17T15:01:05.62493874Z     at ClientDuplexStream._receiveStatus (/var/javascript/node_modules/grpc/src/client.js:233:8)
2017-12-17T15:01:05.624955322Z     at /var/javascript/node_modules/grpc/src/client.js:757:12 code: 8, metadata: Metadata { _internal_repr: {} } }
2017-12-17T15:01:05.62530396Z WARNING cache [PUT] on:status { code: 8,
2017-12-17T15:01:05.625318997Z   details: 'Received message larger than max (15221287 vs. 4194304)',
2017-12-17T15:01:05.625337622Z   metadata: Metadata { _internal_repr: {} } } 

[graphicore]

For GitHub PRs its probably good advice to have a reasonable upper limit, since anybody can make PR's to GitHub, this is easy to use in a malicious way or also to just by accident cause harm.

[davelab6]

Shouldn't the dispatcher be able to drop a large message, and carry on?

[graphicore]

Of course, the error must be caught and handled gracefully. The question is how large a message can be. There's a cap on the gRPC side, obviously, but we still need to control the max size for our needs. E.g. 15 MB is easy with CJK-Families. I'll catch and handle the error today (printing to stderr, so we get a report in the management console and don't forget about it). Later, it's probably necessary to implement chunked messages for the cache and still maintain an upper limit.

[graphicore]

Actually, looking at that log now, and at the code, the exception is properly handled, and we also have ~40 reports in http://35.192.167.39/dashboard for the GitHub-GoogleFonts/pulls Source. We also have 48 reported errors, so that must be the amount of skipped tests (some from other sources as well!).

[graphicore]

Below is the list of 29 families in master exceeding the max of 4194304 bytes. taviraj should be the first that we have already tested (and it is) while e.g Rubik is in production tested but not in master .

Our now biggest font has 74443295 ~ 71MiB

I'll try for now to put the message limit at 80 MiB.

4 * 2**20 == 4194304 == 4 MiB
80 * 2**20 == 83886080 == 80MiB

I won't remove the error that is emitted to the logs and I don't have the time now to implement a chunked stream of data (we probably don't need it, see the comment):

grpc/grpc#7927 (comment)

Our intention is to support larger message sizes when it makes sense (so it doesn't even need to be a stop-gap), but to still encourage service developers to consider the impact of message sizes getting out-of-hand.

/fonts $ du -b {ofl,apache,ofl,ufl}/* | sort -n -r | head -n 30
74443295	ofl/seoulhangangcondensed
70502519	ofl/seoulhangang
41929857	ofl/seoulnamsancondensed
37571060	ofl/seoulnamsan
25377888	ofl/nanummyeongjo
23840791	ofl/roundedmplus1c
19808940	ofl/kopubbatang
14060684	ofl/nanumgothic
12381722	ofl/mplus1p
11874432	ofl/lato
10126016	ofl/cwtexming
 9926200	ofl/cwtexkai
 9615198	ofl/jejumyeongjo
 9175304	ofl/cwtexfangsong
 9134188	ofl/nanumgothiccoding
 8666770	ofl/firasanscondensed
 8657376	ofl/firasans
 8653144	ofl/firasansextracondensed
 7694521	ofl/cwtexyen
 7589014	ofl/nanumpenscript
 7523010	ofl/cormorantgaramond
 7500283	ofl/nanumbrushscript
 7387440	ofl/cormorantinfant
 6687440	ofl/jejuhallasan
 6029778	ofl/ebgaramond
 5494784	ofl/rubik
 5279401	ofl/cwtexhei
 4778856	ofl/montserrat
 4503269	ofl/seoulnamsanvertical
 4094316	ofl/taviraj

[graphicore]

Ok, that worked http://35.225.170.228/report/6390a809-99ca-4433-9fbd-4b302bca3be9

[davelab6]

Amazing!