Commits vs. rollbacks

[ch1bo]

Context & versions

When opening a head today we saw a rollback going exactly onto a commit tx and then we remained with some commits pending and nobody tried a collect com transaction.

Issue is now to collect logs.

Steps to reproduce

Maybe hard, but do some commits and have a rollback in between them?

Actual behavior

Head is not getting opened because no collect transaction is posted.

Expected behavior

The head to open normally, even if we see a rollback and forward of commit transactions.

[ch1bo]

My node logs:
sebastian-hydra-node.log
sebastian-cardano-node.log

[v0d1ch]

mainnet.log
cardano-node.log
state.txt

[ch1bo]

A reconstruction of the events according to my logs:

2023-03-21T16:34:58.185974508Z sebastian commits @ 87850207
2023-03-21T16:35:20.63664424Z sasha commits @ 87850229

2023-03-21T16:35:23.876833226Z RolledBackward to 87850207

2023-03-21T16:35:24.091232319Z sasha commits @ 87850230
2023-03-21T16:35:28.289744255Z arnaud commits @ 87850237

We would have expected that arnaud's node does post a collect com after seeing his commit, even in presence of the rollback in between.

Unfortunately we do not have the logs anymore of arnaud's log. A concrete step forward could be to change the HeadLogic so everyone tries to post a collect (which is in fact what is in the spec).

[ch1bo]

Aborting was not possible for me as my node sees my own commit as pending still:

cat /data/hydra-node/state | jq '.contents.pendingCommits '
[
  {
    "vkey": "43d8ac3fb8e39b355005ed546c8fc9cd80e4909cb254835513a542b1c9a2e3ad"
  }
]

sebastian-state.txt

And the validator prevents this as I already committed on chain:

{"failureReason":"ValidationFailedV2 (CekError An error has occurred:  User error:\nThe machine terminated because of an error, either from a bu
       │ ilt-in function or from an explicit use of 'error'.) [\"H03\",\"PT5\"]","redeemerPtr":"RdmrPtr Spend 2","tag":"ScriptFailedInWallet"

[ch1bo]

We have found the culprit:

• The chainState in the headState is "duplicated" and ends up to be not accurately reflecting the slot when the commit happened, i.e. the state corresponding to sebastian's commit would include say slot 87850229

• This is likely due to the fact that we update the chain state in two places

  • Explicitly in the head logic update https://github.com/input-output-hk/hydra/blob/53071b8f9719f76445dc61caa2bd85124cb1f585/hydra-node/src/Hydra/HeadLogic.hs#L501
  • In between callbacks of the chain layer https://github.com/input-output-hk/hydra/blob/53071b8f9719f76445dc61caa2bd85124cb1f585/hydra-node/src/Hydra/Node.hs#L279

• This was done in course of ADR18 ADR 18: Keeping a single state #305.. it seems this requires some more work.