Missing chain updates more easily with persistence

[ffakenz]

Why

As we do persist the chainstate now, we can miss relevant transactions on the L1 quite easily.

For example, our node thinks the head is open, we stop it for some maintenance and while doing so, somebody else closes the head.

In this example, today, we would still assume the head is open, even though we missed the close transaction.

What

We need to ensure to start synchronizing with the L1 from the last persisted point in our chainstate.

[ffakenz]

These are my thoughts on what could be done to solve this problem:

• We need to keep track of the last ChainPoint recordedAt onRollForward
• We are already recording a ChainSlot from a ChainPoint, so maybe we can use replace it instead.
• This replacement will case us to modify the instance IsChainState Tx to make use of the helper function chainSlotFromPoint :: ChainPoint -> ChainSlot.
• Because we already have an option startChainFrom, now it will be conflicting with recordedAt.
• To resolve this conflict, will apply the following rule:
  • fail to start node if startChainFrom > recordedAt
  • use it otherwise if Just
  • use recordedAt if Nothing

[abailly]

Would this be an issue if we persisted the event stream of the node? then we could simply restart the node at the chain point corresponding to the last observed tx and that's it

[ch1bo]

then we could simply restart the node at the chain point corresponding to the last observed tx and that's it

This is exactly what this work item should be doing. Just without an event stream, but with the persisted aggregated state.

---

Quickly discussed this with @ffakenz today:

• We should change the ChainStateAt to contain recordedAt :: ChainPoint
• We shall NOT change the IsChainState type class definition, only update the instance for `ChainStateAt
• We shall provide the last persisted Maybe ChainPoint to the withDirectCchain, which then decides
  • If both, persisted and --start-chain-from are Just, use the smaller (earlier) one.
  • If one is provided (Just), use that one.
  • If both are Nothing, do queryTip like now.