Create framework for safe rollout of back-compatible runtime document schema changes

[vladsud]

Required pre-reading

https://github.com/microsoft/FluidFramework/blob/main/packages/dds/SchemaVersioning.md
Discussion in https://dev.azure.com/fluidframework/internal/_workitems/edit/5699

Problem statement

We are lacking safe way of deploying new runtime capabilities. Our clients will make mistakes with making decisions on when it’s safe to enable new capabilities like op compression / op grouping. When mistakes happen, we want that to be very easy to identify.
- Based on our own experience with Microsoft applications, we know it’s not always the case. For example, prior assert 0x162 was hit in production and a lot of smart engineers were not able to track it down. But as part of testing #20109, I hit it right away when I made a mistake in config and let old runtime version collaborate with new runtime version that had the new features enabled.

Proposed solution (implementation)

Proposed solution adds document schema controller that manages part of the document schema - the part that FluidFramework runtime controls (summaries & op format).

All new capabilities (like op compression) will follow the following lifetime cycle:

1. An ability to write in new schema is added and shipped dark
2. New capability will not be used by runtime (even in presence of # 3 below) until document schema reflects that such capability is required to interpret document.
3. An application, when it is safe for the application, will instruct runtime (through feature gates or code change) to start leveraging new capability.
4. When runtime detects a mismatch between document schema required capabilities and runtime feature set required, it will do the following:
   1. If it’s subtraction (i.e. document schema says - compression is required, but options/configuration instructs not to use compression), the feature will be disabled in current session. This allows application to quickly pullback capabilities that are causing troubles in production. At the same time, document schema will require such capability, as some parts of the document might require it (i.e. old application versions will continue not be able to open such documents)
   2. If it’s an addition (I.e. application wants to use compression, but it’s not referenced in document schema), document schema will be changed to list such new capability (through ops, communicating to all current and future clients). Only after that happened, new capabilities can be used.
      1. Depending on a capability, it might take an effect in future sessions only.
5. All existing document format features should consult controller's current session settings in determining if some feature is enabled or not. They should not consult anything else, including options, feature gates, etc. Those are inputs into controller's calculus only.
6. A “legacy” mode is added for all the applications that were shipping to production preliminary / internal 2.0 bits, and thus were defining document schema implicitly (by using new capabilities). This mode will not generate ops (as part of # 4.2 flow) as such ops would break unsuspecting older clients.
   1. Once these changes ship and saturate to reasonable level, such applications are advised to graduate from legacy mode.

Also see "Implementation details" section at the bottom for deeper details.

Important outcome:
When / If applications enable new capabilities and it happens too early (i.e. before reasonable saturation occurs), old clients will fail (when they see document schema change op) in predictable way, allowing application developers (and FF developers) much easier to diagnose such issues and react to them faster.
This mostly helps 2.0 -> 3.0 future changes in schema though. 1.3-> 2.0 has only partial solution (1.3 will fail to process document schema change ops, but if such ops are summarized, FF 1.3 could continue to limp along).

Implementation details / discussion

As of now, I use “regular” runtime ops to communicate document schema changes. This could be changed in the future to some other mechanism (like quorum). Document schema has a version (version defines expectations around structure that represents document schema, not capabilities of document schema). Version can be bumped in the future to define different operation mode. As any other change, it would need to ship dark and saturate before it can engage, but old clients that do not understand new schema will fail and will not cause eventual consistency issues.

Schemas are changed in Compare-And-Swap manner. Client who wants to change scheme sends a message that essentially says “chance a schema if document schema is A (before change)”. If some other client changed schema right before it, then such operation will fail (be treated as noop).
At the moment, clients do not retry. Expectation is that further sessions will do full recall, and if needed, will propose a new schema.

Schema is serialized in a summary and changes to schema follow eventual consistency rules (i.e. schema can be reliably calculated at any sequence number in lifetime of the document based on ops, or summaries + ops).

[Abe27342]

How are you imagining this bit evolves over time? The handling here is a little surprising to me: I would have thought this would be an area where we add back-compat code for older versions and only throw on 'new' versions we don't understand, e.g. something like if (!supportedDocSchemaVersions.has(documentSchema.version)) { throw } (or even a <= if we went with a number over a string). Of course that approach is equivalent for now since we only support one version.

Based on interface contracts, I would have expected us to generally support

[vladsud]

We can make it complex when we need it. For now, code understands only 1.0, and has no clue what 2.0 means, so it validates it directly.
Or are you asking - should we support some future where 2.0 is back-compatible with 1.0?
It's hard for me to imagine what 2.0 would mean / shape it takes, so I'm not sure I want to build something more complicated today to support it (like a property that tells that even though it's 2.0, 1.0 should proceed with it as if it was 1.0)

[msfluid-bot]

	Warnings
⚠️	Bundle size regression detected -- please investigate before merging!

⯅ @fluid-example/bundle-size-tests: +21.36 KB

Metric Name	Baseline Size	Compare Size	Size Diff
aqueduct.js	514.76 KB	520.09 KB	⯅ +5.33 KB
azureClient.js	605.87 KB	611.22 KB	⯅ +5.35 KB
connectionState.js	680 Bytes	680 Bytes	■ No change
containerRuntime.js	249.19 KB	254.53 KB	⯅ +5.33 KB
fluidFramework.js	340.94 KB	340.94 KB	■ No change
loader.js	127.97 KB	127.97 KB	■ No change
map.js	41.35 KB	41.35 KB	■ No change
matrix.js	143.61 KB	143.61 KB	■ No change
odspClient.js	574.33 KB	579.68 KB	⯅ +5.35 KB
odspDriver.js	97.49 KB	97.49 KB	■ No change
odspPrefetchSnapshot.js	41.91 KB	41.91 KB	■ No change
sharedString.js	161.38 KB	161.38 KB	■ No change
sharedTree.js	331.08 KB	331.08 KB	■ No change
Total Size	3.3 MB	3.32 MB	⯅ +21.36 KB

---

Baseline commit: fb8fb7d

Generated by 🚫 dangerJS against a209315

[andre4i]

LGTM 👍 from my side