Unable to deliver ICA acks when host chain is using cometbft-db v0.13+ and goleveldb

[joe-bowman]

Note: I'm not sure whether this lies in ibc-go, cosmos-sdk (I think this is unlikely), cometbft-db, or hermes (also unlikely), but I have stumbled across a weird issue whereby ICA acks are not being delivered when the host chain is using cometbft-db v0.13.0+.

Using cometbft-db v1.0.1, the packets are sent to the host chain from the controller, and are executed. The acknowledgement is never received. Hermes complains with the following error:

2024-11-27T14:33:26.262295Z  WARN ThreadId(01) relay_ack_packet_messages{src_chain=mocha-4 src_port=icahost src_channel=channel-211 dst_chain=quicksilver-f8f9-1}: no packet data was pulled at height <=4-3355062 for sequences 1..=50, this might be due to the data not being available on the configured endpoint. Please verify that the RPC endpoint has the required packet data, for more details see https://hermes.informal.systems/advanced/troubleshooting/cross-comp-config.html#uncleared-pending-packets

I can query via GRPC or RPC for PacketAcknowledgement(s), and it returns the expected output.

I can replicate this reliably on Quicksilver (ibc-go v6.3.1), Celestia (ibc-go v6.2.2) and Osmosis (ibc-go v8.5.2) - migrating to cometbft-db v0.12.0+ (the only relevant change here is cometbft/cometbft-db#168 which no longer copies the iterator key/value and passes this responsibility to the caller) is causing Hermes to no longer find the acks to send back to the controller!

I can start hermes pointing to v27.0.0 of Osmosis built with v0.14.1 (the version in go.mod) and it won't work; I can rebuild with v0.12.0 and it does work - without even restarting Hermes.

Notably, this only appears to effect goleveldb, and not pebbledb - which I believe why this is not an issue we are seeing in production, because most production relayers are using PebbleDB as the backend.

[gjermundgaraba]

Hi @joe-bowman, sorry for the late reply here. This got lost in the end of year craziness :)

At first glance, it seems likely to be an issue in cometbft-db. Especially if the only differentiating factor is the cometbft-db version and the type of database (and that the packet acknowledgements are being returned by ibc in the grpc query).

I am probably going to close the issue on this side, but would love to hear if you any updates on the issue in the past month first (or if you disagree that this is not an ibc-go issue)