mismatched block state root on mainnet

[0xc0ffeebabeeth]

Describe the bug

The error initally happened after upgrading from reth alpha 9 to 22 at block height ~19426610.

2024-03-17T19:17:39.671899Z WARN Error while processing payload error=Failed to insert block (hash=0x3a813f3d60cbd6d7397a0644fa8b46b0ae1b6a45522b36252e155a33e73f2078, number=19426684, parent_hash=0x84a53b3a4ce399cf351e527ba740dcebfceb3dc4a6f779cda5d381dafc9dd1c1): mismatched block state root: got 0x1b5d26a71f6ba7d79b81e075010572f5d1801511f010a7a528358db484a07deb, expected 0xe57d1da160feb9c06acb28294b12197d32f26706e24aede1cac712335ce9a58d
2024-03-17T19:17:39.671917Z WARN Invalid block error on new payload invalid_hash=0x3a813f3d60cbd6d7397a0644fa8b46b0ae1b6a45522b36252e155a33e73f2078 invalid_number=19426684 error=mismatched block state root: got 0x1b5d26a71f6ba7d79b81e075010572f5d1801511f010a7a528358db484a07deb, expected 0xe57d1da160feb9c06acb28294b12197d32f26706e24aede1cac712335ce9a58d
2024-03-17T19:17:39.671925Z WARN Bad block with hash hash=0x3a813f3d60cbd6d7397a0644fa8b46b0ae1b6a45522b36252e155a33e73f2078 header=Header { parent_hash: 0x84a53b3a4ce399cf351e527ba740dcebfceb3dc4a6f779cda5d381dafc9dd1c1, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5, state_root: 0xe57d1da160feb9c06acb28294b12197d32f26706e24aede1cac712335ce9a58d, transactions_root: 0x3c639b85a8eabe2e98336370d6dedb50f7c963e5c28ef8c5eab31cb10fecb914, receipts_root: 0x001ff93590b3382d71167a2c2aaced4f91e84726499a53bd811594709339c56a, withdrawals_root: Some(0x94d71e38890df56b852e4ad50c75e56c97bba97d74ceb124737fcfcd032aa65e), logs_bloom: 0x356d7587c15b4121530a2691ba60d3e5114f1126080602f8a2950503f68aea9bf253811e069dc40b4258da3b093cc13c83f5c9248c61ec75c6b785cc8eee4999181bf4dfd35b1a6918eb722df57928f483be021f6c771b0825261cc79bac0a967b2a2910972346295cccddd74878eded12780259d7c6448dd8c35ef0112d41560fa12af52085686c40638ed3034b0b3230300025fd8426c9fa7ec4e80c3c8448fb7a9f75c5e779cf0b2cf3d4bc1685b99fe79c0f6a5624d7992d5068b4ae85f452194f16143f889c915c24a0c17f89091060627635ae009669d4addbf5d4f98a0ab9382ba7b260d2c347978185cc9095dca87d9764ae24ebc12946252fca3c5e, difficulty: 0x0_U256, number: 19426684, gas_limit: 30000000, gas_used: 13104299, timestamp: 1710339299, mix_hash: 0x7f68d4b81eed6594dda06860072d4fa57429976bb438f463bb0dc293d1b509f9, nonce: 0, base_fee_per_gas: Some(60808654744), blob_gas_used: Some(0), excess_blob_gas: Some(262144), parent_beacon_block_root: Some(0xbf71cc5c66c595e3708a341c6f5f0c52c2df75ea16e17696f09ea16bcde9a624), extra_data: 0x6265617665726275696c642e6f7267 }

After downgrading to alpha 21 it initally seemed to move on just fine with syncing, but subsequently failed at 19426745:

2024-03-17T19:27:36.714690Z WARN Error while processing payload error=Failed to insert block (hash=0x6c7c26c7a2459354bd83fa3c1729fd975f57d744200342c22a112b296393fa4e, number=19426745, parent_hash=0xa4837855b2cf626fe24ef4a9d3f7db4becadff03c9ad6da79d91abb312106bd8): mismatched block state root: got 0xa20a7cb4d1bc937a2b931584f6961986afa08a252304915bf8a4d5e2b65647d4, expected 0x53001e79ec26066e770235ee92989a9313766f6fafd61641c751cbf1be0b50dc
2024-03-17T19:27:36.714709Z WARN Invalid block error on new payload invalid_hash=0x6c7c26c7a2459354bd83fa3c1729fd975f57d744200342c22a112b296393fa4e invalid_number=19426745 error=mismatched block state root: got 0xa20a7cb4d1bc937a2b931584f6961986afa08a252304915bf8a4d5e2b65647d4, expected 0x53001e79ec26066e770235ee92989a9313766f6fafd61641c751cbf1be0b50dc
2024-03-17T19:27:36.714716Z WARN Bad block with hash hash=0x6c7c26c7a2459354bd83fa3c1729fd975f57d744200342c22a112b296393fa4e header=Header { parent_hash: 0xa4837855b2cf626fe24ef4a9d3f7db4becadff03c9ad6da79d91abb312106bd8, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5, state_root: 0x53001e79ec26066e770235ee92989a9313766f6fafd61641c751cbf1be0b50dc, transactions_root: 0x00be89819d150908d554a8ad513e8c3b11e78ab3c98c38e442851b70c6796086, receipts_root: 0xd9b60a55536f7bfd86c6329abe90d584e0f696058bc1457b6324f306607f7cc1, withdrawals_root: Some(0x42788a0109c3884ef232c33aa14b388b3a37694dec45540e3e4b2f1770ee0f84), logs_bloom: 0x5bbf756fe9ebe9533769a0befbe99859fd464afa1e7710eda68f027ff42ff3d7b5dfddfadc99f66fcfb2ff8f7f3793782fe7f9589ffebfe7bf7faf9abbbfbbfdf953fe7f57d7effffb6f79d9ddceb5ebcc7b57fd3ef70f5269b7ffddc2ff6f6633db5ccbaffffab21b3dffff575fffd1bcd5b37b349e2faedb57f2966e9fafd37c59d6fd97297f4dfbd96fe753ba87fd57bdfc35d79c7fbe44727dfbfbfbf52f7efc61ebff8bf9fbf706dbfebf785ddc677d9ffbf67ee50a8f3b68663f3d5264e5b9dcff773acfe375d63983db7fe755bffcbf9fa6e94ab7736f57c75becfc7d7eff784d261faedcbf95d6fecbb7dfdfeb5995fbb3ef6d534d699d139fe77e75, difficulty: 0x0_U256, number: 19426745, gas_limit: 30000000, gas_used: 25233181, timestamp: 1710340055, mix_hash: 0x8ae47369d5f16718ecb3c5277c2f92c2bd035cfe989847f50f48eb24bdfa52fc, nonce: 0, base_fee_per_gas: Some(59506106079), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x99d22474c0d4b3ac1dff8abca102dd3764ae8070a5a31a646177d9164415286c), extra_data: 0x6265617665726275696c642e6f7267 }

Steps to reproduce

1. Sync reth alpha 9 to height ~19426610
2. Switch over to reth alpha 22
3. Continue syncing
4. Run into the error

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

Official reth-v0.1.0-alpha.21, reth-v0.1.0-alpha.22 and reth-v0.1.0-alpha.9 builds from github releases.

What database version are you on?

Nuked the box and resynced with reth beta, presumably the default one of reth 9 since thats what was initally synced with.

What type of node are you running?

Archive (default)

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[argakiig]

I can add to this as well. I have a reth alpha 22 instance that recently ran across this

2024-03-25T00:33:13.761392Z  WARN Unwinding{stage=MerkleUnwind}: sync::stages::merkle: Failed to verify block state root target_block=19500975 got=0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b expected=SealedHeader { header: Header { parent_hash: 0x09ee7bb66dffcdb8505591e60e2a698255a4b0e4a2cbf9f0f8e7ecda27a36530, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97, state_root: 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7, transactions_root: 0x7b84a0519ab2e51d9cbcfc495b8048f935b28830e748c859561ec6c0b5af3f6c, receipts_root: 0x2b8d487859b5515370245db0e56be0004ebb3fb5e5b1043e5eb751e34c56107b, withdrawals_root: Some(0x7b9e97cf3c7f47f14f1c90f04315a6273c5c5ce16c7fc30ed1907fbe501c102c), logs_bloom: 0x49692301400f0068544c29e7a01ba10d01882e422dc3908045291923182bc2ac2db12d3a347520491005124804200b980a21c426bc01382805902900403ea34528cc8d4daa000c0b6a83e10b1add2229480b0993e47d190105c48df2b07c82089b3160030b738f95ce9cf408c4604c49661c86e88c1704050d4610983609056530400f7a2ac54140004bf7a9342010f784808c35ddc98e2d4867204f4194b0211ef2117485ee64467aae51d80cf2ee081d86a05a8020008905fc04fd912212301c3617a202c25ce21102f0110110569924190310c01a5cb518032413464e66701cf12b282b142035a90449936743e909358d57077a3925781139381213807641, difficulty: 0x0_U256, number: 19500975, gas_limit: 30000000, gas_used: 10198858, timestamp: 1711241603, mix_hash: 0x065661613341b44ef5da561f4cb0cbdabc822f1b400615fbf3a8a409d648c47c, nonce: 0, base_fee_per_gas: Some(16543282452), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x98348d1831e88d9fcaed95c52671f39f577b19c5a2fd4e128ffbb2f98f4a0f95), extra_data: 0x546974616e2028746974616e6275696c6465722e78797a29 }, hash: 0xf1ebd37de131638fd776bbe24b768ab2092385cb14845c4236d5c8d92da65810 }
2024-03-25T00:33:13.821028Z ERROR reth::cli: shutting down due to error
Error: stage encountered an error in block #19500975: validation error: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7

Caused by:
   0: validation error: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7
   1: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7

Location:
    /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39

using db format 1 as this is still alpha

that particular instance will run for approx 8 min and then die due to the above.

[pires]

Was anyone here able to move forward without deleting the entire data dir and start over?

[Rjected]

I can add to this as well. I have a reth alpha 22 instance that recently ran across this

2024-03-25T00:33:13.761392Z  WARN Unwinding{stage=MerkleUnwind}: sync::stages::merkle: Failed to verify block state root target_block=19500975 got=0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b expected=SealedHeader { header: Header { parent_hash: 0x09ee7bb66dffcdb8505591e60e2a698255a4b0e4a2cbf9f0f8e7ecda27a36530, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97, state_root: 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7, transactions_root: 0x7b84a0519ab2e51d9cbcfc495b8048f935b28830e748c859561ec6c0b5af3f6c, receipts_root: 0x2b8d487859b5515370245db0e56be0004ebb3fb5e5b1043e5eb751e34c56107b, withdrawals_root: Some(0x7b9e97cf3c7f47f14f1c90f04315a6273c5c5ce16c7fc30ed1907fbe501c102c), logs_bloom: 0x49692301400f0068544c29e7a01ba10d01882e422dc3908045291923182bc2ac2db12d3a347520491005124804200b980a21c426bc01382805902900403ea34528cc8d4daa000c0b6a83e10b1add2229480b0993e47d190105c48df2b07c82089b3160030b738f95ce9cf408c4604c49661c86e88c1704050d4610983609056530400f7a2ac54140004bf7a9342010f784808c35ddc98e2d4867204f4194b0211ef2117485ee64467aae51d80cf2ee081d86a05a8020008905fc04fd912212301c3617a202c25ce21102f0110110569924190310c01a5cb518032413464e66701cf12b282b142035a90449936743e909358d57077a3925781139381213807641, difficulty: 0x0_U256, number: 19500975, gas_limit: 30000000, gas_used: 10198858, timestamp: 1711241603, mix_hash: 0x065661613341b44ef5da561f4cb0cbdabc822f1b400615fbf3a8a409d648c47c, nonce: 0, base_fee_per_gas: Some(16543282452), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x98348d1831e88d9fcaed95c52671f39f577b19c5a2fd4e128ffbb2f98f4a0f95), extra_data: 0x546974616e2028746974616e6275696c6465722e78797a29 }, hash: 0xf1ebd37de131638fd776bbe24b768ab2092385cb14845c4236d5c8d92da65810 }
2024-03-25T00:33:13.821028Z ERROR reth::cli: shutting down due to error
Error: stage encountered an error in block #19500975: validation error: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7

Caused by:
   0: validation error: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7
   1: mismatched block state root: got 0xfc9482c650b36f5d975144cecd4d53dbbd570b5972e4ddb8c80d8abe2aba7a1b, expected 0xfe1bd7ff16cd049ad18fe636bcab85b1fb479e1aef9490ed75cb9dec69f2a9c7

Location:
    /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/task/poll.rs:255:39

using db format 1 as this is still alpha

that particular instance will run for approx 8 min and then die due to the above.

Hi @argakiig / @pires are you still experiencing state root mismatch? if so, could you send the most recent ~50-100 lines of failure logs, and the full output of reth db stats?

[argakiig]

I have not had the issue in quite a bit thankfully

[Rjected]

Good to hear! @0xc0ffeebabeeth forgot to tag you in the above comment - mind providing that info if you're still experiencing this?

[gakonst]

Fixed in #7753.