Metastore Recovery on Command Topic Failure

[agavra]

Is your feature request related to a problem? Please describe.

In a recent production issue, the command topic was lost due to user-error (setting the retention of the topic to 0) and we were unable to recover the data and recreate the queries.

Describe the solution you'd like

We should have some lightweight, last-ditch mechanism to recover from loss of the command topic. One solution is to persist the metastore/command-topic to a file with a checksum and use that to manually re-issue statements if necessary.

Describe alternatives you've considered

Open to suggestions.

[blueedgenick]

if you've failed to secure the environment against this kind of mistake, and are concerned about it happening, there are general solutions to this ? e.g. use Connect to sink the command topic to a file in s3

[agavra]

closing as duplicate of #5575