Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

erigon polygon hangs @[1/9 Snapshots] progress=100.00% after restart #11210

Closed
g45g4wwf opened this issue Jul 17, 2024 · 4 comments
Closed

erigon polygon hangs @[1/9 Snapshots] progress=100.00% after restart #11210

g45g4wwf opened this issue Jul 17, 2024 · 4 comments

Comments

@g45g4wwf
Copy link

g45g4wwf commented Jul 17, 2024 •
edited
Loading

System information

Erigon version: 3 git_tag=v2.59.1-4953-g1b1079926c

OS & Version: Linux

Commit hash: 1b10799

Erigon Command (with flags/config):
erigon
--chain=bor-mainnet
--datadir=/data
--port=31303
--bor.heimdall=http://heimdall:1317
--authrpc.addr=0.0.0.0
--authrpc.port=8651
--authrpc.jwtsecret=/secrets/jwt.hex
--authrpc.vhosts=any
--http.addr=0.0.0.0
--http.vhosts=*
--http.corsdomain=*
--http.api=admin,eth,net,web3,debug,engine,erigon,bor
--ws
--ws.port=8646
--private.api.addr=0.0.0.0:9091
--torrent.download.rate=64mb
--torrent.port=42069

Consensus Layer: heimdall (0xpolygon/heimdall:1.0.6)

Consensus Layer Command (with flags/config):
start --home=/heimdall-home --chain=mainnet --rest-server --eth_rpc_url=http://erigon:8545

Chain/Network: polygon main

Expected behaviour

erigon syncs

Actual behaviour

Apparently, there was a problem causing Erigon to restart during the sync.
After the restart, it returned to stage 1/9 and has been stuck at 100% progress for several hours now.

Steps to reproduce the behaviour

Backtrace

erigonpolygon  | [INFO] [07-17|02:10:12.203] [snapshots:blocks:retire] Stat           blocks=58999k indices=58999k alloc=22.9GB sys=53.2GB
erigonpolygon  | [INFO] [07-17|02:10:12.956] [snapshots:bor:retire] Stat              blocks=58999k indices=58999k alloc=23.2GB sys=53.2GB
erigonpolygon  | [INFO] [07-17|02:10:21.600] [snapshots:blocks:retire] Stat           blocks=59000k indices=59000k alloc=29.8GB sys=53.2GB
erigonpolygon  | [INFO] [07-17|02:10:34.727] [5/9 Bodies] Downloading block bodies    block_num=59010407 delivery/sec=3.9MB wasted/sec=4.0MB remaining=4614 delivered=1586 blk/sec=52 cache=256.0MB alloc=21.1GB sys=53.2GB
erigonpolygon  | [INFO] [07-17|02:11:04.729] [5/9 Bodies] Downloading block bodies    block_num=59010613 delivery/sec=2.9MB wasted/sec=3.2MB remaining=4408 delivered=2798 blk/sec=93 cache=256.0MB alloc=35.0GB sys=53.2GB
erigonpolygon  | [INFO] [07-17|02:11:23.052] logging to file system                   log dir=/data/logs file prefix=erigon log level=info json=false
erigonpolygon  | [INFO] [07-17|02:11:23.053] Build info                               git_branch=main git_tag=v2.59.1-4914-g8324145f0a git_commit=8324145f0acb6809184032b98f98829c92254619
erigonpolygon  | [INFO] [07-17|02:11:23.053] 
erigonpolygon  | 	########b          oo                               d####b. 
erigonpolygon  | 	##                                                      '## 
erigonpolygon  | 	##aaaa    ##d###b. dP .d####b. .d####b. ##d###b.     aaad#' 
erigonpolygon  | 	##        ##'  '## ## ##'  '## ##'  '## ##'  '##        '## 
erigonpolygon  | 	##        ##       ## ##.  .## ##.  .## ##    ##        .## 
erigonpolygon  | 	########P dP       dP '####P## '#####P' dP    dP    d#####P 
erigonpolygon  | 	                           .##                              
erigonpolygon  | 	                       d####P                               
erigonpolygon  | 		 
erigonpolygon  | [INFO] [07-17|02:11:23.053] Starting Erigon on Bor Mainnet... 
erigonpolygon  | [INFO] [07-17|02:11:23.058] Maximum peer count                       ETH=100 total=100
erigonpolygon  | [INFO] [07-17|02:11:23.058] starting HTTP APIs                       port=8545 APIs=admin,eth,net,web3,debug,engine,erigon,bor
erigonpolygon  | [INFO] [07-17|02:11:23.059] torrent verbosity                        level=WRN
erigonpolygon  | [INFO] [07-17|02:11:25.064] Set global gas cap                       cap=50000000
erigonpolygon  | [INFO] [07-17|02:11:25.067] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=64mb upload.rate=4mb
erigonpolygon  | [INFO] [07-17|02:11:25.138] Opening Database                         label=chaindata path=/data/chaindata
erigonpolygon  | [INFO] [07-17|02:11:25.147] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
erigonpolygon  | [INFO] [07-17|02:11:25.151] Initialised chain configuration          config="{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor}" genesis=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b
erigonpolygon  | [INFO] [07-17|02:12:43.556] Initialising Ethereum protocol           network=137
erigonpolygon  | [INFO] [07-17|02:12:43.557] Opening Database                         label=bor path=/data/bor
erigonpolygon  | setup_dxb:18034 opening after an unclean shutdown, but boot-id(bf194d59abfdefb9-59146ef6083345c0) is MATCH: rollback NOT needed, steady-sync NEEDED
erigonpolygon  | [INFO] [07-17|02:12:44.161] Starting private RPC server              on=0.0.0.0:9091
erigonpolygon  | [INFO] [07-17|02:12:44.161] new subscription to logs established 
erigonpolygon  | [INFO] [07-17|02:12:44.161] rpc filters: subscribing to Erigon events 
erigonpolygon  | [INFO] [07-17|02:12:44.161] new subscription to newHeaders established 
erigonpolygon  | [INFO] [07-17|02:12:44.162] New txs subscriber joined 
erigonpolygon  | [INFO] [07-17|02:12:44.163] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=[::]:8646 http.url=[::]:8545
erigonpolygon  | [WARN] [07-17|02:12:44.229] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
erigonpolygon  | [INFO] [07-17|02:12:44.232] [1/9 Snapshots] Requesting downloads 
erigonpolygon  | [WARN] [07-17|02:12:44.237] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
erigonpolygon  | [INFO] [07-17|02:12:44.250] [snapshots] no progress yet              files=2 list=v1-058900-059000-headers.seg,v1-058900-059000-bodies.seg
erigonpolygon  | [INFO] [07-17|02:12:44.255] Started P2P networking                   version=67 self=enode://775cbe19bdbf115b9af6a063e6edba895875e21d42273bebadae10a691d6c5bf188549ebad38a4f972a93d739af454cded091c0e109ccd7b8b79ca99e6779935@127.0.0.1:30303 name=erigon/v3.00.0-dev-8324145f/linux-amd64/go1.22.4
erigonpolygon  | [INFO] [07-17|02:12:44.262] Started P2P networking                   version=68 self=enode://775cbe19bdbf115b9af6a063e6edba895875e21d42273bebadae10a691d6c5bf188549ebad38a4f972a93d739af454cded091c0e109ccd7b8b79ca99e6779935@127.0.0.1:31303 name=erigon/v3.00.0-dev-8324145f/linux-amd64/go1.22.4
erigonpolygon  | [INFO] [07-17|02:12:44.324] [txpool] Started 
erigonpolygon  | [INFO] [07-17|02:13:03.595] [snapshots] no progress yet              files=2 list=v1-058900-059000-headers.seg,v1-058900-059000-bodies.seg
erigonpolygon  | [INFO] [07-17|02:13:04.251] [1/9 Snapshots] downloading header-chain progress="100.00% 3.7TB/3.7TB" time-left=999hrs:99m total-time=20s download=0B/s upload=0B/s peers=1 files=1353 metadata=1353/1353 connections=1 alloc=19.0GB sys=19.5GB
erigonpolygon  | [INFO] [07-17|02:13:23.593] [snapshots] no progress yet              files=2 list=v1-058900-059000-headers.seg,v1-058900-059000-bodies.seg
erigonpolygon  | [INFO] [07-17|02:13:24.251] [1/9 Snapshots] downloading header-chain progress="100.00% 3.7TB/3.7TB" time-left=999hrs:99m total-time=40s download=0B/s upload=0B/s peers=2 files=1353 metadata=1353/1353 connections=2 alloc=19.1GB sys=19.7GB
@AskAlexSharov
Copy link
Collaborator

we will investigate.

workaround:
if "just restart" doesn't help
try rm -rf datadir/downloader
erigon will re-verify all local files and likely continue

@g45g4wwf
Copy link
Author

thx, but after deleting datadir/downloader/* the download of snapshots restarts completely

erigonpolygon  | [INFO] [07-18|13:40:46.804] logging to file system                   log dir=/data/logs file prefix=erigon log level=info json=false
erigonpolygon  | [INFO] [07-18|13:40:46.804] Build info                               git_branch=main git_tag=v2.59.1-4953-g1b1079926c git_commit=1b1079926ce17a8439b0e86e8bde7c2bbc13b061
erigonpolygon  | [INFO] [07-18|13:40:46.804] 
erigonpolygon  | 	########b          oo                               d####b. 
erigonpolygon  | 	##                                                      '## 
erigonpolygon  | 	##aaaa    ##d###b. dP .d####b. .d####b. ##d###b.     aaad#' 
erigonpolygon  | 	##        ##'  '## ## ##'  '## ##'  '## ##'  '##        '## 
erigonpolygon  | 	##        ##       ## ##.  .## ##.  .## ##    ##        .## 
erigonpolygon  | 	########P dP       dP '####P## '#####P' dP    dP    d#####P 
erigonpolygon  | 	                           .##                              
erigonpolygon  | 	                       d####P                               
erigonpolygon  | 		 
erigonpolygon  | [INFO] [07-18|13:40:46.804] Starting Erigon on Bor Mainnet... 
erigonpolygon  | [INFO] [07-18|13:40:46.805] Maximum peer count                       ETH=100 total=100
erigonpolygon  | [INFO] [07-18|13:40:46.805] starting HTTP APIs                       port=8545 APIs=admin,eth,net,web3,debug,engine,erigon,bor
erigonpolygon  | [INFO] [07-18|13:40:46.805] torrent verbosity                        level=WRN
erigonpolygon  | [INFO] [07-18|13:40:48.808] Set global gas cap                       cap=50000000
erigonpolygon  | [INFO] [07-18|13:40:48.809] [Downloader] Running with                ipv6-enabled=true ipv4-enabled=true download.rate=128mb upload.rate=4mb
erigonpolygon  | [INFO] [07-18|13:40:48.809] Opening Database                         label=chaindata path=/data/chaindata
erigonpolygon  | [INFO] [07-18|13:40:48.818] [db] open                                label=chaindata sizeLimit=12TB pageSize=8192
erigonpolygon  | [INFO] [07-18|13:40:48.819] Initialised chain configuration          config="{ChainID: 137, Homestead: 0, DAO: <nil>, Tangerine Whistle: 0, Spurious Dragon: 0, Byzantium: 0, Constantinople: 0, Petersburg: 0, Istanbul: 3395000, Muir Glacier: 3395000, Berlin: 14750000, London: 23850000, Arrow Glacier: <nil>, Gray Glacier: <nil>, Terminal Total Difficulty: <nil>, Merge Netsplit: <nil>, Shanghai: <nil>, Cancun: <nil>, Prague: <nil>, Osaka: <nil>, Engine: bor}" genesis=0xa9c28ce2141b56c474f1dc504bee9b01eb1bd7d1a507580d5519d4437a97de1b
erigonpolygon  | [INFO] [07-18|13:41:57.077] Initialising Ethereum protocol           network=137
erigonpolygon  | [INFO] [07-18|13:41:57.077] Opening Database                         label=bor path=/data/bor
erigonpolygon  | [INFO] [07-18|13:41:57.191] Starting private RPC server              on=0.0.0.0:9091
erigonpolygon  | [INFO] [07-18|13:41:57.191] new subscription to logs established 
erigonpolygon  | [INFO] [07-18|13:41:57.191] rpc filters: subscribing to Erigon events 
erigonpolygon  | [INFO] [07-18|13:41:57.191] new subscription to newHeaders established 
erigonpolygon  | [INFO] [07-18|13:41:57.192] New txs subscriber joined 
erigonpolygon  | [INFO] [07-18|13:41:57.192] JsonRpc endpoint opened                  ws=true ws.compression=true grpc=false websocket.url=[::]:8646 http.url=[::]:8545
erigonpolygon  | [INFO] [07-18|13:41:57.196] [txpool] Started 
erigonpolygon  | [WARN] [07-18|13:41:57.225] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
erigonpolygon  | [WARN] [07-18|13:41:57.229] NAT ExternalIP resolution has failed, try to pass a different --nat option err="no UPnP or NAT-PMP router discovered"
erigonpolygon  | [INFO] [07-18|13:41:57.236] [1/9 Snapshots] Requesting downloads 
erigonpolygon  | [INFO] [07-18|13:41:57.249] Started P2P networking                   version=67 self=enode://775cbe19bdbf115b9af6a063e6edba895875e21d42273bebadae10a691d6c5bf188549ebad38a4f972a93d739af454cded091c0e109ccd7b8b79ca99e6779935@127.0.0.1:30303 name=erigon/v3.00.0-dev-1b107992/linux-amd64/go1.22.4
erigonpolygon  | [INFO] [07-18|13:41:57.254] Started P2P networking                   version=68 self=enode://775cbe19bdbf115b9af6a063e6edba895875e21d42273bebadae10a691d6c5bf188549ebad38a4f972a93d739af454cded091c0e109ccd7b8b79ca99e6779935@127.0.0.1:31303 name=erigon/v3.00.0-dev-1b107992/linux-amd64/go1.22.4
erigonpolygon  | [INFO] [07-18|13:41:57.345] [snapshots] no progress yet              files=1353 list=v1-041500-042000-borevents.seg,v1-055400-055500-transactions.seg,idx/v1-code.1600-1664.ef,idx/v1-logtopics.1408-1472.ef,history/v1-code.0-64.v,...
erigonpolygon  | [INFO] [07-18|13:42:17.262] [snapshots] no progress yet              files=1298 list=v1-029000-029500-borevents.seg,v1-033000-033500-borspans.seg,v1-057300-057400-borevents.seg,idx/v1-accounts.1984-2048.ef,idx/v1-code.1984-2048.ef,...
erigonpolygon  | [INFO] [07-18|13:42:17.346] [1/9 Snapshots] downloading header-chain progress="0.01% 365.0MB/3.7TB" time-left=58hrs:45m total-time=20s download=18.2MB/s upload=0B/s peers=1 files=1353 metadata=1353/1353 connections=1 alloc=23.1GB sys=23.7GB

@AskAlexSharov
Copy link
Collaborator

ah, i didn't notice it's erigon3. so, it's known problem: #10976 here is the fix: #11227

@AskAlexSharov
Copy link
Collaborator

merged #11227

you need:

  • git pull the main
  • rm -rf datadir/downloader
    it will fix your problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AskAlexSharov @g45g4wwf