[Ropsten] Deep chain reorg breaks chain processor #17227

ccpwcn · 2018-07-23T03:56:04Z

Hi there,

please note that this is an issue tracker reserved for bug reports and feature requests.

For general questions please use the gitter channel or the Ethereum stack exchange at https://ethereum.stackexchange.com.

System information

Geth version: Geth/v1.8.7-stable-66432f38/linux-amd64/go1.10.1
OS & Version: CentOS 7

Expected behaviour

Synchronize the data on the test network to my network node succeessfully

Actual behaviour

has a lot of errors:

INFO [07-23|03:47:22] Imported new chain segment               blocks=1  txs=11  mgas=4.067  elapsed=65.598ms   mgasps=61.996  number=3692495 hash=71f4f3…1f7da3 cache=7.81mB
INFO [07-23|03:47:22] Upgrading chain index                    type=bloombits percentage=0
INFO [07-23|03:47:22] Commit new mining work                   number=3692496 txs=0   uncles=0 elapsed=313.386µs
ERROR[07-23|03:47:22] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:47:24] Generating DAG in progress               epoch=124 percentage=22 elapsed=10m1.912s
INFO [07-23|03:48:05] Generating DAG in progress               epoch=124 percentage=23 elapsed=10m43.689s
INFO [07-23|03:48:35] Imported new chain segment               blocks=1  txs=3   mgas=0.453  elapsed=83.161ms   mgasps=5.442   number=3692496 hash=782320…6e9368 cache=7.77mB
INFO [07-23|03:48:35] Upgrading chain index                    type=bloombits percentage=0
INFO [07-23|03:48:35] Commit new mining work                   number=3692497 txs=8   uncles=0 elapsed=34.338ms
ERROR[07-23|03:48:35] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:48:42] Imported new chain segment               blocks=1  txs=110 mgas=4.709  elapsed=394.823ms  mgasps=11.926  number=3692497 hash=367f8c…5869f8 cache=7.83mB
INFO [07-23|03:48:42] Commit new mining work                   number=3692498 txs=0   uncles=0 elapsed=705.589µs
ERROR[07-23|03:48:43] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:48:47] Generating DAG in progress               epoch=124 percentage=24 elapsed=11m24.944s
INFO [07-23|03:48:55] Imported new chain segment               blocks=1  txs=35  mgas=1.473  elapsed=130.021ms  mgasps=11.327  number=3692498 hash=6c341a…183382 cache=7.89mB
INFO [07-23|03:48:55] Upgrading chain index                    type=bloombits percentage=0
INFO [07-23|03:48:55] Commit new mining work                   number=3692499 txs=0   uncles=0 elapsed=634.375µs
ERROR[07-23|03:48:56] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:48:56] Imported new chain segment               blocks=1  txs=8   mgas=0.420  elapsed=90.273ms   mgasps=4.650   number=3692499 hash=9b534b…9a0a48 cache=7.83mB
INFO [07-23|03:48:56] Commit new mining work                   number=3692500 txs=0   uncles=0 elapsed=494.562µs
ERROR[07-23|03:48:57] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:49:29] Generating DAG in progress               epoch=124 percentage=25 elapsed=12m7.132s
INFO [07-23|03:49:40] Imported new chain segment               blocks=1  txs=69  mgas=2.726  elapsed=409.938ms  mgasps=6.650   number=3692500 hash=158f25…c7b105 cache=7.88mB
INFO [07-23|03:49:40] Upgrading chain index                    type=bloombits percentage=0
INFO [07-23|03:49:40] Commit new mining work                   number=3692501 txs=1   uncles=0 elapsed=1.071ms
ERROR[07-23|03:49:40] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [07-23|03:49:47] Imported new chain segment               blocks=1  txs=15  mgas=0.580  elapsed=44.275ms   mgasps=13.105  number=3692501 hash=f8f5ce…ee5f6f cache=7.89mB
INFO [07-23|03:49:47] Commit new mining work                   number=3692502 txs=1   uncles=0 elapsed=769.893µs
ERROR[07-23|03:49:48] Section processing failed                type=bloombits error="chain reorged during section processing"

Steps to reproduce the behaviour

geth application started as command-line:

nohup geth --testnet --fast --rpc --rpccorsdomain "*" --datadir data --mine --minerthreads=2 --txpool.lifetime 24h0m0s --txpool.globalqueue 1048576 -cache=2048 --port 30303 --rpcapi "db,eth,net,web3,personal" --etherbase="0xC0AE3D68Ce5df82aD0594B547Db6D5b2fbEFfaEd" > nohup.out &

IMPORTENT INFORMATION

Geth application has no errors when started, but this a lot of error appeared when the data that has been synchronized size more than 25GB, I tried the action many times, and I have the same error.

The text was updated successfully, but these errors were encountered:

karalabe · 2018-07-23T08:59:26Z

Ping @zsfelfoldi and perhaps @rjl493456442 (since you've been looking into bloombits lately).

tanhuiya · 2018-10-29T07:20:41Z

same issue on privete chain
and getbalance return 0x0

Geth/v1.8.17-stable-8bbe7207/linux-amd64/go1.11.1

brandoncurtis · 2019-02-17T19:56:52Z

Geth/v1.8.22-stable/linux-amd64/go1.11

This started happening when upgrading from Geth 1.8.20 to 1.8.22.

I know I can fix the problem by deleting the entire chain and starting over, but resyncing this thing takes a week (and this is a pretty terrible option on a piece of infrastructure in production) and I'd like to help solve this bug if at all possible.

See also #18215, #15301, #15169.

$ geth --cache 1024 --v5disc --nousb --syncmode full --gcmode archive --testnet --port 30323
...
INFO [02-17|11:52:04.841] Imported new chain segment               blocks=1 txs=4  mgas=0.238 elapsed=12.718ms  mgasps=18.715  number=5038488 hash=894c5d…c5f306 cache=0.00B
INFO [02-17|11:52:04.844] Upgrading chain index                    type=bloombits percentage=98
ERROR[02-17|11:52:04.845] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [02-17|11:52:08.509] Imported new chain segment               blocks=1 txs=14 mgas=2.934 elapsed=37.524ms  mgasps=78.179  number=5038489 hash=3b2d04…3a8cb5 cache=0.00B
ERROR[02-17|11:52:08.511] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [02-17|11:52:19.905] Imported new chain segment               blocks=1 txs=1  mgas=0.021 elapsed=8.414ms   mgasps=2.496   number=5038490 hash=ce07e0…bda0bc cache=0.00B
INFO [02-17|11:52:19.909] Upgrading chain index                    type=bloombits percentage=98
ERROR[02-17|11:52:19.913] Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [02-17|11:52:24.977] Imported new chain segment               blocks=1 txs=3  mgas=0.162 elapsed=11.517ms  mgasps=14.086  number=5038491 hash=432c45…75a83a cache=0.00B
ERROR[02-17|11:52:24.977] Section processing failed                type=bloombits error="chain reorged during section processing"
```

And again with debug:
```
$ geth --cache 1024 --v5disc --nousb --syncmode full --gcmode archive --testnet --port 30323 --debug
INFO [02-17|11:56:12.040|core/blockchain_insert.go:74]     Imported new chain segment               blocks=1  txs=0  mgas=0.000 elapsed=10.425ms  mgasps=0.000  number=5038519 hash=f3c5c4…328c30 cache=0.00B
ERROR[02-17|11:56:12.043|core/chain_indexer.go:341]        Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [02-17|11:56:12.534|core/blockchain_insert.go:74]     Imported new chain segment               blocks=1  txs=3  mgas=0.886 elapsed=21.668ms  mgasps=40.868 number=5038520 hash=c295ef…13f40e cache=0.00B
ERROR[02-17|11:56:12.536|core/chain_indexer.go:341]        Section processing failed                type=bloombits error="chain reorged during section processing"
INFO [02-17|11:56:15.192|core/blockchain_insert.go:74]     Imported new chain segment               blocks=1  txs=0  mgas=0.000 elapsed=9.267ms   mgasps=0.000  number=5038521 hash=69c969…001d94 cache=0.00B
INFO [02-17|11:56:15.192|core/chain_indexer.go:321]        Upgrading chain index                    type=bloombits percentage=98
ERROR[02-17|11:56:15.193|core/chain_indexer.go:341]        Section processing failed                type=bloombits error="chain reorged during section processing"
```

mmitech · 2019-02-18T13:45:21Z

I am having the same issue:

Feb 18 13:43:55 geth[47696]: INFO [02-18|13:43:55.535] Imported new chain segment               blocks=1 txs=0 mgas=0.000 elapsed=12.349ms mgasps=0.000 number=5043662 hash=40d91a…6d5237 age=1m20s cache=4.03kB
Feb 18 13:43:55 geth[47696]: INFO [02-18|13:43:55.536] Upgrading chain index                    type=bloombits percentage=98
Feb 18 13:43:55 geth[47696]: ERROR[02-18|13:43:55.537] Section processing failed                type=bloombits error="chain reorged during section processing"
Feb 18 13:43:57 geth[47696]: INFO [02-18|13:43:57.497] Imported new chain segment               blocks=2 txs=6 mgas=0.479 elapsed=177.750ms mgasps=2.692 number=5043664 hash=2927b0…e0b528 age=1m11s cache=74.01kB
Feb 18 13:43:57 geth[47696]: ERROR[02-18|13:43:57.499] Section processing failed                type=bloombits error="chain reorged during section processing"
Feb 18 13:43:58 geth[47696]: INFO [02-18|13:43:58.182] Imported new chain segment               blocks=3 txs=26 mgas=2.983 elapsed=286.131ms mgasps=10.425 number=5043667 hash=5b602c…f808cd cache=257.94kB
Feb 18 13:43:58 geth[47696]: ERROR[02-18|13:43:58.183] Section processing failed                type=bloombits error="chain reorged during section processing"
Feb 18 13:43:59 geth[47696]: INFO [02-18|13:43:59.168] Imported new chain segment               blocks=1 txs=24 mgas=1.470 elapsed=198.954ms mgasps=7.389  number=5043668 hash=11937e…c6de04 cache=372.48kB
Feb 18 13:43:59 geth[47696]: ERROR[02-18|13:43:59.169] Section processing failed                type=bloombits error="chain reorged during section processing"

ekryski · 2019-03-01T17:40:55Z

Same thing. On Ropsten. Running v1.8.23.

harnen · 2019-03-01T19:30:37Z

Same here, using private blockchain and geth v1.9.0-unstable-7240f4d8

aliaksandr-adzinets · 2019-03-07T12:51:19Z

I was getting bad block errors on Geth 1.8.21 (Ropsten), so I updated to 1.8.23.
Now I'm getting this buddy:

ERROR[03-07|12:58:44.596] Section processing failed
type=bloombits error="canonical block #5148672 unknown"

Full re-sync?

cyberfab007 · 2019-03-07T21:06:58Z

Same here

steel1041 · 2019-03-11T07:09:02Z

same here private chain:Geth/v1.8.23-stable-c9427004/linux-amd64/go1.10.4
ERROR[03-11|15:06:43.320] Section processing failed type=bloombits error="chain reorged during section processing"

karalabe · 2019-03-12T10:25:16Z

Could anyone please paste a longer dump of logs? These here contain the symptom, but not the root cause. We have a hunch it was a deep reorg that triggers it, but that should be also logged in the logs. If we could have the vicinity of the logs where this issue first starts to happen, that would be really helpful.

princesinha19 · 2019-04-05T18:52:43Z

I am getting this error too.

cyberfab007 · 2019-04-06T04:14:26Z

In the end it was a corrupt chain

SirSteven1 · 2019-05-09T02:08:47Z

same here

carlosgub · 2019-05-13T02:18:09Z

INFO [05-13|02:17:19.837] Commit new mining work =0 gas=0 fees=0 elapsed=184.311µs INFO [05-13|02:17:25.967] Successfully sealed new block f80e69 elapsed=6.129s
INFO [05-13|02:17:25.967] 🔗 block reached canonical chain INFO [05-13|02:17:25.967] 🔨 mined potential block INFO [05-13|02:17:25.967] Commit new mining work =0 gas=0 fees=0 elapsed=171.161µs ERROR[05-13|02:17:25.968] Section processing failed on processing"
INFO [05-13|02:17:37.722] Successfully sealed new block 3aa837 elapsed=11.754s INFO [05-13|02:17:37.722] 🔗 block reached canonical chain INFO [05-13|02:17:37.722] 🔨 mined potential block INFO [05-13|02:17:37.722] Upgrading chain index ERROR[05-13|02:17:37.722] Section processing failed on processing"
INFO [05-13|02:17:37.722] Commit new mining work =0 gas=0 fees=0 elapsed=188.419µs
INFO [05-13|02:17:53.641] Successfully sealed new block a69fe4 elapsed=15.918s
INFO [05-13|02:17:53.641] 🔗 block reached canonical chain INFO [05-13|02:17:53.641] 🔨 mined potential block INFO [05-13|02:17:53.641] Upgrading chain index INFO [05-13|02:17:53.642] Commit new mining work =0 gas=0 fees=0 elapsed=216.233µs
ERROR[05-13|02:17:53.642] Section processing failed on processing" INFO [05-13|02:17:56.697] Successfully sealed new block c1ac23 elapsed=3.055s
INFO [05-13|02:17:56.697] 🔗 block reached canonical chain INFO [05-13|02:17:56.698] 🔨 mined potential block INFO [05-13|02:17:56.698] Commit new mining work =0 gas=0 fees=0 elapsed=178.174µs ERROR[05-13|02:17:56.698] Section processing failed on processing number=16744 sealhash=0fb7ee…723f2f uncles=0 txs
│
number=16744 sealhash=0fb7ee…723f2f hash=725ff4…│
number=16737 hash=14c4e5…ee57c8
number=16744 hash=725ff4…f80e69
number=16745 sealhash=3504d9…0cdcc9 uncles=0 txs│
│
type=bloombits error="chain reorged during secti
number=16745 sealhash=3504d9…0cdcc9 hash=44fddb…│
│
number=16738 hash=e6beae…a5f59f
number=16745 hash=44fddb…3aa837
type=bloombits percentage=50 │
type=bloombits error="chain reorged during secti│
number=16746 sealhash=d18d34…0c336a uncles=0 txs
number=16746 sealhash=d18d34…0c336a hash=668bf2…│
number=16739 hash=b11eec…2ef1db
number=16746 hash=668bf2…a69fe4
type=bloombits percentage=50 │
number=16747 sealhash=f32271…32c99f uncles=0 txs│
type=bloombits error="chain reorged during secti
│
number=16747 sealhash=f32271…32c99f hash=41185f…│
number=16740 hash=87d839…be5ab3
number=16747 hash=41185f…c1ac23
number=16748 sealhash=f29daf…bd8ef1 uncles=0 txs│
│
type=bloombits error="chain reorged during secti

this is my log

I use this for start geth

geth --datadir ethdata --networkid 1234 --rpc --rpcport 8545 --rpcapi "db,net,
web3,personal,eth,web3,miner" --rpccorsdomain "*" --rpcaddr "0.0.0.0"

my genesis.json

{                                                                                                                   
                                                                                                                
    "config":{                                                                                                      
        "chainId":15,                                                                                               
        "homesteadBlock":0,                                                                                         
        "eip155Block":0,                                                                                            
        "eip158Block":0                                                                                             
    },                                                                                                              
    "difficulty":"400",                                                                                             
    "gasLimit":"2100000000",                                                                                        
    "alloc":{                                                                                                       
    }                                                                                                               
                                                                                                                
}

karalabe · 2019-07-02T13:41:57Z

This is supposedly fixed on master.

rjl493456442 self-assigned this Jul 23, 2018

karalabe changed the title ~~chain reorged during section processing~~ [Ropsten] Deep chain reorg breaks chain processor Mar 12, 2019

adamschmideg mentioned this issue May 14, 2019

Section processing failed error="chain reorged during section processing" #15301

Closed

adamschmideg assigned zsfelfoldi and unassigned rjl493456442 May 14, 2019

adamschmideg mentioned this issue May 14, 2019

Bloombits: "chain reorged during section processing", index upgrade stuck at 0 percent complete #15169

Closed

adamschmideg added this to the 1.9.0 milestone May 14, 2019

zsfelfoldi mentioned this issue Jun 20, 2019

core: fix chain indexer reorg bug #19748

Merged

karalabe closed this as completed Jul 2, 2019

mrFawkes mentioned this issue Sep 16, 2020

error="chain reorged during section processing" ava-labs/coreth#31

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ropsten] Deep chain reorg breaks chain processor #17227

[Ropsten] Deep chain reorg breaks chain processor #17227

ccpwcn commented Jul 23, 2018

karalabe commented Jul 23, 2018

tanhuiya commented Oct 29, 2018 •

edited

Loading

brandoncurtis commented Feb 17, 2019 •

edited

Loading

mmitech commented Feb 18, 2019

ekryski commented Mar 1, 2019

harnen commented Mar 1, 2019

aliaksandr-adzinets commented Mar 7, 2019

cyberfab007 commented Mar 7, 2019

steel1041 commented Mar 11, 2019

karalabe commented Mar 12, 2019

princesinha19 commented Apr 5, 2019

cyberfab007 commented Apr 6, 2019

SirSteven1 commented May 9, 2019

carlosgub commented May 13, 2019 •

edited

Loading

karalabe commented Jul 2, 2019

[Ropsten] Deep chain reorg breaks chain processor #17227

[Ropsten] Deep chain reorg breaks chain processor #17227

Comments

ccpwcn commented Jul 23, 2018

System information

Expected behaviour

Actual behaviour

Steps to reproduce the behaviour

IMPORTENT INFORMATION

karalabe commented Jul 23, 2018

tanhuiya commented Oct 29, 2018 • edited Loading

brandoncurtis commented Feb 17, 2019 • edited Loading

mmitech commented Feb 18, 2019

ekryski commented Mar 1, 2019

harnen commented Mar 1, 2019

aliaksandr-adzinets commented Mar 7, 2019

cyberfab007 commented Mar 7, 2019

steel1041 commented Mar 11, 2019

karalabe commented Mar 12, 2019

princesinha19 commented Apr 5, 2019

cyberfab007 commented Apr 6, 2019

SirSteven1 commented May 9, 2019

carlosgub commented May 13, 2019 • edited Loading

karalabe commented Jul 2, 2019

tanhuiya commented Oct 29, 2018 •

edited

Loading

brandoncurtis commented Feb 17, 2019 •

edited

Loading

carlosgub commented May 13, 2019 •

edited

Loading