Fatal panic: state service commit block failed: verified checkpoints must be committed transactionally: CommitFinalized(Closed)

[dconnolly]

Caught by Sentry: https://sentry.io/organizations/zfnd/issues/2354800484/?project=5540918&query=is%3Aunresolved&statsPeriod=14d

Error

state service commit block failed: verified checkpoints must be committed transactionally: CommitFinalized(Closed)

Analysis

We've seen this bug a few times now, but we need to be able to reproduce it to diagnose it and fix it.

Looking at recent commits, there are a few potential causes:

• new RocksDB version in Update dependencies to support Apple M1, fix Windows CI LLVM paths #2026
• bugs from duplicate dependencies introduced by Update dependencies to support Apple M1, fix Windows CI LLVM paths #2026 or other recent changes
• recent fixes to the timeout and mutex code
• interrupt handler bugs like Interrupt handler does not work when a blocking task is running #1351, if RocksDB starts shutting down before the shutdown flag is set

More broadly, it could be a regression in:

• rustc atomic codegen or stdlib
• rustc async codegen or stdlib
• tokio
• rocksdb
• Zebra

It's probably a regression in PR #1637, which fixed ticket #1576.

Possible Fixes

Potential diagnostic:

• Add restart acceptance tests #1666 - restart acceptance tests

Potential fixes:

• Ignore some panics during shutdown using the panic handler #1677 - move shutdown panic skipping code into the panic handler
• Interrupt handler does not work when a blocking task is running #1351 - interrupt handler does not work when a blocking task is running
• Design and implement graceful shutdown for Zebra #1678 - design and implement graceful shutdown (large)

Metadata

key	value
version	1.0.0-alpha.6
git commit	afac2c2
Zcash network	Mainnet
location	/zebra/zebra-consensus/src/checkpoint.rs:923:18

Backtrace

Backtrace:
   0: backtrace::capture::Backtrace::create
   1: backtrace::capture::Backtrace::new

[teor2345]

Looking at recent commits, there are a few potential causes:

• new RocksDB version in Update dependencies to support Apple M1, fix Windows CI LLVM paths #2026
• bugs from duplicate dependencies introduced by Update dependencies to support Apple M1, fix Windows CI LLVM paths #2026 or other recent changes
• recent fixes to the timeout and mutex code

[teor2345]

@dconnolly I think I'll need you to give me access to the Google Cloud logs for the following instance_ids:

• zebrad-main-5bhl
• zebrad-main-qbg2

Or if the logs have been deleted already, let's increase the retention interval.

[teor2345]

It's also possible that something like #1351 is the cause, if RocksDB starts shutting down before the shutdown flag is set.

[teor2345]

Unfortunately, there's no backtrace in sentry, so we can't really diagnose this issue any further.

It's also likely that the backtrace wouldn't provide any useful info - what we really need is the logs, so we can see what the rest of the app was doing.

[teor2345]

Here's another possible cause…

We could be exiting inside the state service (debug_stop_at_height) or a panic, and racing with checkpointer shutdown.

Here are some possible fixes:

• make the debug_stop_at_height code call a zebra exit function that sets the shutdown bool first
• Replace all instances of exit with Zebra exit
• Set the is_shutting_down flag as early as possible in the panic handler
• Register an at_exit handler that sets the is_shutting_down flag

This work could also be done in #1678.

[teor2345]

There's not much we can do here until we get better diagnostics, and it seems to have been a transient bug.

[mpguerra]

lowering priority to Medium for now also

[teor2345]

There's not much we can do here until we get better diagnostics, and it seems to have been a transient bug.

We've seen this bug again, but we need to be able to reproduce it to diagnose it and fix it.

[teor2345]

We haven't seen this bug for a while, marking it as low priority.

[teor2345]

We haven't seen this bug for a while, and we've added a shutdown workaround. So I think we can close this ticket.