Hermes unable to read acks on goleveldb node after #168

[joe-bowman]

After upgrading above v0.12.0 of cometbft-db, and using goleveldb as the backend, Hermes is no longer able to query IBC packet data (specifically, acknowledgements, in this case). This is immediately resolved by downgrading back to v0.12.0 and restarting the node (so it is not a genuine data availability issue, despite presenting as such).

Note: I'm not sure whether this lies in ibc-go, cosmos-sdk (I think this is unlikely), cometbft-db, or hermes (also unlikely), but I have stumbled across a weird issue whereby ICA acks are not being delivered when the host chain is using cometbft-db v0.13.0+.

I can replicate this reliably on Quicksilver (ibc-go v6.3.1), Celestia (ibc-go v6.2.2) and Osmosis (ibc-go v8.5.2) - migrating to cometbft-db v0.12.0+ (the only relevant change here is #168 which no longer copies the iterator key/value and passes this responsibility to the caller) is causing Hermes to no longer find the acks to send back to the controller! This suggests the underlying issue is resulting from the upgrade to v0.13+ of cometbft-db (even if the fix might be in the ibc-go channelKeeper query logic; hence raised here: cosmos/ibc-go#7605).

Notably, this only appears to effect goleveldb, and not pebbledb - which I believe why this is not an issue we are seeing in production, because most production relayers are using PebbleDB as the backend. I am wondering whether the change in #168 is behaving differently under goleveldb and pebbledb.

I would appreciate your thoughts.