Block large payloads inside DocumentDeltaConnection, size due to the 1MB Kafka limit

[andre4i]

See #7599

It would block message transmission and close the container but only with explicit feature gate.

[msfluid-bot]

⯅ @fluid-example/bundle-size-tests: +3.03 KB

Metric Name	Baseline Size	Compare Size	Size Diff
container.js	169.6 KB	169.82 KB	⯅ +219 Bytes
map.js	47.21 KB	47.21 KB	■ No change
matrix.js	143.43 KB	143.43 KB	⯅ +1 Bytes
odspDriver.js	186.55 KB	189.29 KB	⯅ +2.74 KB
odspPrefetchSnapshot.js	41.28 KB	41.36 KB	⯅ +76 Bytes
sharedString.js	164.24 KB	164.24 KB	⯅ +1 Bytes
Total Size	784.99 KB	788.02 KB	⯅ +3.03 KB

---

Baseline commit: 646a987

Generated by 🚫 dangerJS against b66f6e0

[andre4i]

@markfields @vladsud please take a look at this. There is a bit of a behavior change with regards to retrying on error in the deltamanager.

[andre4i]

^^ We would always retry, regardless of the type of error. Not sure if this has always been intentional or not..

[vladsud]

Here is the story :)
In ODSP, we definitely want to reconnect always, because we may get disconnect with 403 due to token expiration. 403 is critical error (i.e., in general it's a game over event), but across layers we always do one retry with refreshed token to ensure host has a chance to provide new token.

With that said, I think that's the wrong layer to participate in this game. I.e. ODSP has to project such errors as recoverable. I'd need to look at the code to say if it's the case. And not sure about FRS.

Also, worth noting that for long period of time we consider errors without canRetry to be recoverable. We changed that (maybe 6 months back) - any exception, whether it's a bug in our code, or in host code (like token callback) is catastrophic.

So, to summarize, this code is likely wrong. But I do not think you are making it any more right :)

[andre4i]

I don't believe there is a viable way right now for propagating non-retriable errors from the driver to the container. Maybe a TTL on the error itself? What are your thoughts?

[vladsud]

I know you looked into it before, but I want to poke more.
Is there really no way to determine that socket.io disconnect happened due to size violation?
Is there anything we can do to observe lower-level traffic to figure it out? That would be obviously much better direction to convert such errors to catastrophic errors (not sure about 0.9Mb limit).

[andre4i]

Is there really no way to determine that socket.io disconnect happened due to size violation?

None that I can find :( #8179 (comment) was the farthest I've gotten investigating/debugging this particular issue. I think the best way is to actually add the limitation to the server (our server) based on my notes here: #7599 (comment), push that value down using the client config, the documentdeltaconnection should read that config and either block OR if we fix the 1MB kafka limit do the fix in the runtime layer to support larger batches.

The current PR is to be explicit about a limit that is 'de facto' breaking things silently.

[vladsud]

I think this comment suggest that what we measure is not what actually gets counted by socket.io. I.e. if socket.io strigifies payload, then it will add all these escape characters and they will go against the limit, right?

[andre4i]

This approach is currently not preferred, as the overhead for each OP risks making this a performance bottleneck. We'll be exploring alternative solutions, such as a socket.io limitation along with an improvement of the error retry mechanism, the latter also fixing the current reconnect loop on socket.io error issue.