Panic: Inconsistent state after reverting block

[viquezclaudio]

While syncing a full node to the testnet I faced the following panic:

2024-05-28T19:46:29.016482000Z ERROR panic                | thread 'tokio-runtime-worker' panicked at 'assertion `left == right` failed: Inconsistent state after reverting block #5548738:MI:0b49bdfcef - Micro(MicroBlock { header: MicroHeader { network: TestAlbatross, version: 1, block_number: 5548738, timestamp: 1716925588499, parent_hash: 1a630615c9981915cfaf01a27083fc34565b8a204967cea6803647d0d138ea2a, seed: VrfSeed("600fa70d01a3faf13c2a4f0fcd675d897c1c875a9a9d7d6c45ec2a2308594f53f6496d7c0769c6a09f7f50991f9807287daf3b2268f47f1b79f4be1daf000d03eaabd715046575dff05eb292eab52de9292b637651e083740ceca6d1656d3002"), extra_data: [], state_root: ee1ec3f9b2fd8d26ae660ee710cf28fb7eeaa909eb375c6779f3809d1d9924c8, body_root: c7210ab42f122f701369b3fa4c4d154680b03ae3f33a803f1991e3480ca692a9, diff_root: 368b906056cad3f7fbd3572a4e37cba39f4d0fbd3166883d89eb3ee15d43a3e5, history_root: fb1bc627ba9435dc3ee52da300032c33cfaf70a4bc16390c2183328bda682021 }, justification: Some(Micro(Ed25519Signature(ed25519::Signature { R: [130, 234, 91, 146, 137, 49, 254, 153, 239, 84, 254, 12, 42, 117, 206, 181, 154, 220, 129, 135, 166, 42, 157, 111, 158, 104, 206, 99, 94, 191, 31, 120], s: [30, 161, 161, 48, 244, 114, 57, 246, 112, 155, 100, 15, 175, 183, 50, 56, 34, 223, 221, 185, 28, 135, 127, 9, 209, 72, 57, 102, 195, 111, 241, 9] }))), body: Some(MicroBody { equivocation_proofs: [], transactions: [Ok(Transaction { sender: Address("3b07ded6c87e6f67fb49bd1509572e469869f358"), sender_type: Basic, sender_data: [], recipient: Address("db005da56d618fd97e8ab26a017a0bb292efb0fa"), recipient_type: Basic, recipient_data: [], value: Coin(100), fee: Coin(0), validity_start_height: 5548737, network_id: TestAlbatross, flags: TransactionFlags(0x0), proof: [0, 178, 135, 253, 133, 103, 158, 17, 229, 244, 92, 164, 86, 222, 225, 104, 107, 57, 122, 228, 166, 252, 225, 135, 242, 207, 58, 54, 147, 228, 5, 246, 120, 0, 110, 241, 1, 90, 170, 229, 218, 94, 197, 43, 33, 35, 189, 132, 67, 101, 159, 77, 216, 225, 219, 222, 198, 148, 126, 170, 191, 224, 198, 102, 157, 63, 211, 122, 132, 133, 207, 21, 136, 103, 180, 233, 241, 43, 150, 85, 254, 179, 85, 185, 183, 115, 211, 234, 39, 188, 159, 229, 122, 41, 110, 14, 200, 2], valid: false }), Ok(Transaction { sender: Address("46a68f37ccbc8317d1f8d80300c8aec710ca2dd4"), sender_type: Basic, sender_data: [], recipient: Address("4d305f16395cbe6e387e2d79264b7c87c8e99a89"), recipient_type: Basic, recipient_data: [], value: Coin(1), fee: Coin(0), validity_start_height: 5548737, network_id: TestAlbatross, flags: TransactionFlags(0x0), proof: [0, 181, 239, 129, 20, 21, 81, 238, 115, 59, 12, 231, 166, 141, 103, 109, 95, 9, 113, 218, 68, 214, 62, 104, 1, 147, 168, 28, 178, 212, 78, 249, 132, 0, 241, 153, 20, 107, 111, 211, 99, 33, 139, 128, 45, 165, 163, 103, 248, 23, 8, 60, 20, 2, 145, 245, 8, 16, 126, 10, 182, 151, 52, 192, 194, 104, 3, 13, 228, 26, 150, 61, 253, 228, 226, 108, 163, 41, 210, 233, 203, 172, 200, 255, 60, 171, 16, 61, 241, 11, 185, 255, 129, 218, 219, 248, 29, 3], valid: false })] }) })
  left: d9f792292042b14facc2440913e6ef4f9da390cc7a60423fe3ff554632925e37
 right: ee1ec3f9b2fd8d26ae660ee710cf28fb7eeaa909eb375c6779f3809d1d9924c8': blockchain/src/blockchain/rebranch_utils.rs:160
   0: backtrace::capture::Backtrace::new
   1: log_panics::Config::install_panic_hook::{{closure}}
   2: std::panicking::rust_panic_with_hook
   3: std::panicking::begin_panic_handler::{{closure}}
   4: std::sys_common::backtrace::__rust_end_short_backtrace
   5: _rust_begin_unwind
   6: core::panicking::panic_fmt
   7: core::panicking::assert_failed_inner
   8: core::panicking::assert_failed
   9: nimiq_blockchain::blockchain::rebranch_utils::<impl nimiq_blockchain::blockchain::blockchain::Blockchain>::rebranch_to
  10: nimiq_blockchain::blockchain::push::<impl nimiq_blockchain::blockchain::blockchain::Blockchain>::rebranch
  11: nimiq_blockchain::blockchain::push::<impl nimiq_blockchain::blockchain::blockchain::Blockchain>::push_wrapperfn
  12: nimiq_consensus::sync::live::queue::blockchain_push
  13: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  14: tokio::runtime::task::core::Core<T,S>::poll
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::blocking::pool::Inner::run
  17: std::sys_common::backtrace::__rust_begin_short_backtrace
  18: core::ops::function::FnOnce::call_once{{vtable.shim}}
  19: std::sys::unix::thread::Thread::new::thread_start
  20: __pthread_joiner_wake

Full logs are attached (They contain a print of the start key nibbles of state sync, since I found this while testing the log message.)

InconsistentStatePanic.log

[sisou]

The assert in question:

core-rs-albatross/blockchain/src/blockchain/rebranch_utils.rs

Lines 158 to 167 in 34da982

	// Verify accounts hash if the tree is complete or changes only happened in the complete part.
	if let Some(accounts_hash) = self.state.accounts.get_root_hash(Some(write_txn)) {
	assert_eq!(
	prev_info.head.state_root(),
	&accounts_hash,
	"Inconsistent state after reverting block {} - {:?}",
	block,
	block,
	);
	}

[viquezclaudio]

Adding logs without the StateChunkLogs and Block broadcasting logs so they are "cleaner"

InconsistentStatePanic-noChunkLogs.log