Got my internet offline for few minutes and didnt want to get new blocks anymore

[userbox020]

Describe the bug

Im on reth beta3 version, here i share the logs

Steps to reproduce

disconect internet for few minutes, then connect internet again

Node logs

2024-03-22T22:15:06.123886Z  INFO Stage is always skipped
	
	
2024-03-22T22:15:06.123898Z  INFO Preparing stage pipeline_stages=5/12 stage=MerkleUnwind checkpoint=5540373 target=5540373
	
	
2024-03-22T22:15:06.123908Z  INFO Executing stage pipeline_stages=5/12 stage=MerkleUnwind checkpoint=5540373 target=5540373
	
	
2024-03-22T22:15:06.123920Z  INFO Stage finished executing pipeline_stages=5/12 stage=MerkleUnwind checkpoint=5540373 target=5540373 stage_progress=None
	
	
2024-03-22T22:15:06.123934Z  INFO Preparing stage pipeline_stages=6/12 stage=AccountHashing checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.123945Z  INFO Executing stage pipeline_stages=6/12 stage=AccountHashing checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.133702Z  INFO Stage finished executing pipeline_stages=6/12 stage=AccountHashing checkpoint=5540373 target=5540373 stage_progress=100.00%
	
	
2024-03-22T22:15:06.139772Z  INFO Preparing stage pipeline_stages=7/12 stage=StorageHashing checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.139827Z  INFO Executing stage pipeline_stages=7/12 stage=StorageHashing checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.157433Z  INFO Stage finished executing pipeline_stages=7/12 stage=StorageHashing checkpoint=5540373 target=5540373 stage_progress=100.00%
	
	
2024-03-22T22:15:06.163792Z  INFO Preparing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.163822Z  INFO Executing stage pipeline_stages=8/12 stage=MerkleExecute checkpoint=5540370 target=5540373
	
	
2024-03-22T22:15:06.473043Z  WARN Failed to verify block state root target_block=5540373 got=0x8fdfddf2e602ca88c74e9b9127fee5f467b2652484818681599741d5d6a1b331 expected=SealedHeader { hash: 0xd93f4c2c525c2ef9adbf1afc8f94abbd7e224d69eef1e89220163f7c1c943a58, header: Header { parent_hash: 0x4d97484b4434fd717735eea53508736d35ab065bccc405341b9e163e85a3171b, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x0c06b6d4ec451987e8c0b772ffcf7f080c46447a, state_root: 0xc1cf79979b73e93d574a81b1ae1c24489e30c1b3ca31064b7332ab1d589dd9d2, transactions_root: 0xc2c2a0096f224aba096ab331a05e33c12de1f1c74ab8c900f9ec871cfac84e0e, receipts_root: 0xdaea575ec794df77ae9f3bde401a378a51b066e362c73e370cbf5763fdfb4c55, withdrawals_root: Some(0x0b6e8a2a812d74618a842c66943d49ea555d6e6cf16aa6ad0b7fbdd93f354b1e), logs_bloom: 0xc60f6a0e90006298315486bb182a8f22a039d009015c213792dd4aa647ce919240592c65a0607426e9333243110c02ae041a507e9af9f144b7da7b653b35a2194f62b3aa6e252f28ea5e08cb563d8315b620a50583e66640d80bb0418d23723b29f62323feede368d942057512494b2fcd9a85e4496ac6455244aa5fc6189a34dee6c38fb31164286600756072a8f71c8130d467e86594e4b393b5c846e6b8a2629829b2e9c48a5665d05150401c043c5a5b0c16a488835c4d850672154149015a8f530286ce0a308aa960399ab3104403f4e0135f7ee31a9a219304f867fa0876f6a2146832800ba27a302223a5851d9561441b4bfe045a081a5af02f2840c0, difficulty: 0x0_U256, number: 5540373, gas_limit: 30000000, gas_used: 29625426, timestamp: 1711142880, mix_hash: 0xafdfeb84f79b8915aaf1d33d2aaad7d21e238182a01754c670d35af6dfd825d9, nonce: 0, base_fee_per_gas: Some(16), blob_gas_used: Some(786432), excess_blob_gas: Some(82837504), parent_beacon_block_root: Some(0x2a048a4210745a3ee262a9924ba6fd27f71c4575c6f8e3baa490cbeb8e1d0b9f), extra_data: 0x3863333934646239 } }
	
	
2024-03-22T22:15:06.473145Z ERROR Stage encountered a validation error: mismatched block state root: got 0x8fdfddf2e602ca88c74e9b9127fee5f467b2652484818681599741d5d6a1b331, expected 0xc1cf79979b73e93d574a81b1ae1c24489e30c1b3ca31064b7332ab1d589dd9d2 stage=MerkleExecute bad_block=5540373
	
	
2024-03-22T22:15:06.480013Z  INFO Unwinding{stage=StorageHashing}: Stage unwound stage=StorageHashing unwind_to=5540370 progress=5540370 done=true
	
	
2024-03-22T22:15:06.513122Z  INFO Unwinding{stage=AccountHashing}: Stage unwound stage=AccountHashing unwind_to=5540370 progress=5540370 done=true
	
	
2024-03-22T22:15:06.648596Z  WARN Unwinding{stage=MerkleUnwind}: Failed to verify block state root target_block=5540370 got=0x672b9815fc8848cd33372139ce705e95850fe3ffa4b1397a3a2724d5f64fb983 expected=SealedHeader { hash: 0x6cfbabe70b3b7f3744f4be02934471fc3eb8eff502810e9b0678b99a5b596b16, header: Header { parent_hash: 0xd1f3980f1c995a1b56a9e46ae6325b90f75ccccbea4d95e855d432529994b04f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x3826539cbd8d68dcf119e80b994557b4278cec9f, state_root: 0xad69a07c10bbc46b6ea1bf796611cedb7940b47ac4fca38840a6e20d79f619ff, transactions_root: 0x5f8217c165a54d735efe8d22d95a5af8530a9b5d7c5bfea7312ca60257e1e7ff, receipts_root: 0x2e5ff87edb920e758b9c4e880e10103cc1018ff27ceb775c83bca7e26f60c1a4, withdrawals_root: Some(0xb7ffffd961ca1836a618b82dcb1e80e501288074fd6b19fd1e0df60a95729901), logs_bloom: 0xc40409ca883045980150c0221900172880305080c0350966981440021082c24a20080040a20000248120511010b56082c0c0024e0aa21014c59038004f212000403010004ca50f88c296001ad754c09daeac492706026048d8008a4080074213993002c0a64d6610104251401209cf2ecc380ac0414a86040040903908080e010ec245080b684c0163023c225222679330304847404554e14199a18804442c014288400081648e2650c440820110b00928150743a088003c1ce51622974941800194d10a00aa2850e0280199b81b18240220021119494040220491448c212630c456004250008000936e102a20a5841094084410c9bc42f2060a3ba00a280011, difficulty: 0x0_U256, number: 5540370, gas_limit: 30000000, gas_used: 8791592, timestamp: 1711142820, mix_hash: 0x566ada4b499a1f2327141856959f1c5776aeff0cb9c08c20b23bc30da0024126, nonce: 0, base_fee_per_gas: Some(17), blob_gas_used: Some(393216), excess_blob_gas: Some(82706432), parent_beacon_block_root: Some(0x9c9baf178901fd641adfe9dc82f5a9e21783bc67891009528fe784a3b9333aaf), extra_data: 0x } }
	
	
2024-03-22T22:15:06.649349Z ERROR shutting down due to error
	
	
Error: stage encountered an error in block #5540370: validation error: mismatched block state root: got 0x672b9815fc8848cd33372139ce705e95850fe3ffa4b1397a3a2724d5f64fb983, expected 0xad69a07c10bbc46b6ea1bf796611cedb7940b47ac4fca38840a6e20d79f619ff
	
	

	
	
Caused by:
	
	
   0: validation error: mismatched block state root: got 0x672b9815fc8848cd33372139ce705e95850fe3ffa4b1397a3a2724d5f64fb983, expected 0xad69a07c10bbc46b6ea1bf796611cedb7940b47ac4fca38840a6e20d79f619ff
	
	
   1: mismatched block state root: got 0x672b9815fc8848cd33372139ce705e95850fe3ffa4b1397a3a2724d5f64fb983, expected 0xad69a07c10bbc46b6ea1bf796611cedb7940b47ac4fca38840a6e20d79f619ff
	
	

	
	
Location:
	
	
    /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39
	
	
2024-03-22T22:15:06.649892Z  INFO Wrote network peers to file peers_file="/var/lib/reth/known-peers.json"

Platform(s)

Linux (x86)

What version/commit are you on?

beta3

What database version are you on?

v2

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[tobernguyen]

Got this error today too. Reth was never able to recover from that (reth will get into the same error after restarting). Debug logs

reth-crash-2023-03-26.log.zip

[Rjected]

@tobernguyen @userbox020 are you still experiencing this? If so, could you provide the most recent failure logs as well as the full output of reth db stats?

[userbox020]

Solved with version beta5