Persistent cosmic-swingset state

[mhofman]

What is the Problem Being Solved?

Cosmic-swingset has access to a "host" section of the swing-store to keep some state across blocks. However to simplify state-sync (#3769), the implementation in #7225 removed all such host storage that wasn't participating in hangover inconsistency handling, and is not part of the exported data. Only the blockHeight and chainSends performed in the previous block are now kept in the swing-store host storage, and a state-sync restore just repopulates the blockHeight from the snapshot info, and chainSends are empty because state-sync arranges to restore a state not suffering from hangover.

All other state cosmic-swingset currently keeps is transient and can be safely discarded.

However we find ourselves in need of persisting some state again across blocks in cosmic-swingset. In the future, we may want to keep contextual information associated to some of the swingset execution (#7875) or provide a unified persistent storage capability for the host part of a device (e.g. if the host starts recording timers wakeup, see #7846).

This cosmic-swingset state storage needs to be resilient against hangover inconsistency issues, and be captured in and restored from state-sync snapshots.

Description of the Design

TBD

Security Considerations

Making sure any hangover cannot introduce divergence.

Scaling Considerations

TBD

Test Plan

We should actually exercise hangover logic in a test, possibly by forcing a node in a multi-validator setup to rollback the cosmos state by 1 block, or simulate a state-sync when host state is kept.

Upgrade Considerations

TBD

[mhofman]

Conceptually there are 2 existing storages where we can keep cosmic-swingset state: cosmos DB or swing-store.

If we're keeping state in cosmos DB, we have the following constraints:

• read state "synchronously" from the cosmos golang process while processing blocking sends
  • if we want to continue execution between blocks, we need to arrange a local cache that is synchronized back after, like explored by feat(cosmic-swingset): execute during voting time #6741 for the action queue.
• record changes to that state with other chainSends so that a hangover can re-apply changes to the cosmos DB

If we're keeping state in the swing-store, we have the following constraints:

• copy state changes into the cosmos DB along other swing-store "export data", and restore it during swing-store import
• Separate it from the current "host" section which keeps hangover handling state and is not exported to the cosmos DB
  • However similarly to other streamed swing-store "export data", these changes must be replayed in case of hangover

I believe the latter may be simpler and provide more flexibility long term, while still exercising existing pathways. We just need to designate some part of the host storage as part of the swing-store export/import process.