Eta calculator fixes

[daniel-savu]

Bug 1

Closes #2723

The relayer panic is caused by an overflow, bc of dividing by ~6.540888459481895e-211. On my local, the effective rate of indexing starts at 0.61.

{"timestamp":"2023-09-15T09:57:10.746276Z","level":"INFO","fields":{"message":"~~~ blocks_processed: 2508, tip_progression: 2042, elapsed: 757.10340475, old_rate: Some(0.6155037701275111), effective_rate: 0.6155037701275111"},"target":"hyperlane_base::contract_sync::eta_calculator","span":{"domain":"solanadevnet","label":"gas_payments","name":"ContractSync"},"spans":[{"domain":"solanadevnet","label":"gas_payments","name":"ContractSync"}]}

But then both the blocks_processed and the tip_progression are consistently zero, which makes the new_rate be zero (

hyperlane-monorepo/rust/hyperlane-base/src/contract_sync/eta_calculator.rs

Line 41 in eea423a

let new_rate = (blocks_processed - tip_progression) / elapsed;

), and over time it takes over the entire moving average window to make it almost zero, leading to an overflow. 15 mins after that initial log, the effective rate already became 0.00038.

The reason for blocks_processed and tip_progression consistently being zero after the first log is that eta_calculator.calculate(from, tip) is always called with the same from and tip although it expects to get the latest values.

The fix

the tip wasn't being set after the sequence_and_tip query here:

hyperlane-monorepo/rust/hyperlane-base/src/contract_sync/cursor.rs

Line 565 in eea423a

let (count, tip) = self.indexer.sequence_and_tip().await?;

And then the to and from are calculated based on it:

hyperlane-monorepo/rust/hyperlane-base/src/contract_sync/cursor.rs

Line 550 in eea423a

let to = u32::min(

So even though the sync_state internal variables were kept up-to-date, the min(...) would cause the issue.

The first PR commit fixes this.

Bug 2

There was another bug in the eta calculator, caused by it only using next_block to approximate synced state, which doesn't apply to sequence indexing. The way the eta calculator is called had to be changed to use abstract measures of sync progression (which could be blocks or sequences).

The second PR commit fixes this, afaict.

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (7ab7cf5) 64.16% compared to head (7046cbf) 64.16%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2735   +/-   ##
=======================================
  Coverage   64.16%   64.16%           
=======================================
  Files          92       92           
  Lines        1398     1398           
  Branches      186      186           
=======================================
  Hits          897      897           
  Misses        494      494           
  Partials        7        7           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mattiekat]

Lgtm