Geth missing blocks, rewind goes to genesis instead of last good block

[caseybrown89]

System information

Geth version:

/ # geth version
Geth
Version: 1.11.0-unstable
Git Commit: 468d1844c7a32b51eebce6c5f35c44a66b9acf64
Git Commit Date: 20220919
Architecture: amd64
Go Version: go1.18.6
Operating System: linux
GOPATH=
GOROOT=go

CL client & version: gcr.io/prysmaticlabs/prysm/beacon-chain:v3.1.0

OS & Version:

uname -a
Linux geth-mainnet-eth-etl-0 5.4.209-116.367.amzn2.x86_64 #1 SMP Wed Aug 31 00:09:52 UTC 2022 x86_64 Linux

Commit hash : 468d1844c7a32b51eebce6c5f35c44a66b9acf64

Expected behaviour

Over the past couple of months I have been running/syncing a --syncmode=full Geth node on EKS. Recently I was looking at block data and noticed that I am missing data for block 15,012,712 through 15,012,866. I'm not sure how this happened. There have been some unclean shutdowns of Geth in the past but I don't think these took place around the time those blocks were synced.

Missing blocks:

curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xe51367",true],"id":1}' 10.0.143.11:8545 | jq .'result'.'hash'
"0x78db5823c2610aed6b73c28fbc05f4d7b6948787bad33f92c81776bc424f3524"
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xe51368",true],"id":1}' 10.0.143.11:8545 | jq .'result'.'hash'
null
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xe513b5",true],"id":1}' 10.0.143.11:8545 | jq .'result'.'hash'
null
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xe51402",true],"id":1}' 10.0.143.11:8545 | jq .'result'.'hash'
null
curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0xe51403",true],"id":1}' 10.0.143.11:8545 | jq .'result'.'hash'
"0x238e84b4a80403b1425b6dc1e45dd67456dea30e9e84b84b19130ae4eab03fe4"

The log doesn't output anything when I make getBlock calls for missing blocks. It does output some info when trying to get the logs for those blocks:

curl -s -X POST -H 'Content-Type: application/json' -d '{"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock":"0xe51368","toBlock":"0xe51368"}],"id":1}' 10.0.143.11:8545
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to get logs for block #15012712 (0x64940d..45c4aa)"}}
...
ERROR[11-02|14:36:01.487] Missing body but have receipt            hash=64940d..45c4aa number=15,012,712
WARN [11-02|14:36:01.490] Served eth_getLogs                       conn=10.0.15.200:58362 reqid=1 duration=6.981005ms err="failed to get logs for block #15012712 (0x64940d..45c4aa)"

In an attempt to get data for those blocks, I looked for possible ways to fill in the gap. That didn't seem possible so I opted to "rewind" back to the point of the first missing block and then resync from there forward. To do this, I did the following:

• Restart Geth with --maxpeers=0
• Restart Prysm with --no-discovery
• Rewind to the block via

/ # geth attach /data/geth.ipc
Welcome to the Geth JavaScript console!

instance: Geth/v1.11.0-unstable-468d1844-20220919/linux-amd64/go1.18.6
coinbase: 0xb2e19d23405f1b16a86c0d1ad4110e59357bd47c
at block: 15870557 (Mon Oct 31 2022 20:54:35 GMT+0000 (UTC))
 datadir: /data
 modules: admin:1.0 debug:1.0 engine:1.0 eth:1.0 ethash:1.0 miner:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

To exit, press ctrl-d or type exit
> debug.setHead('0xe51368')

At this point, I expected Geth to slowly make its way back to block 15,012,712. I'd then undo the peer/discovery changes and start syncing again.

Actual behaviour

The setHead command ran for a few hours before this appeared in the log:

ERROR[11-01|19:28:24.906] Missing block in the middle, aiming genesis number=15,012,866 hash=096e67..35c886
WARN [11-01|19:28:24.908] Force rewinding till ancient limit       head=0

go-ethereum/core/blockchain.go

Lines 608 to 609 in 2415911

	log.Error("Missing block in the middle, aiming genesis", "number", newHeadBlock.NumberU64()-1, "hash", newHeadBlock.ParentHash())
	newHeadBlock = bc.genesisBlock

After that, disk continued to slowly free up and then stopped when the setHead command returned with null. In total, disk usage went from something like 2.1TB on disk to 1.6TB. At that point, I restarted Geth with peers and resyncs began from block 0. The sync does appear to be going fast-ish as it has made its way to 4MM and change ~12 hours later, but it is showing signs of slowing down.

Steps to reproduce the behaviour

Hard to reproduce. I have EBS snapshots I could offer.

Backtrace

Restarting Geth, executing setHead command and waiting, and restarting Geth:

> debug.setHead('0xe51368')

...

INFO [11-01|17:45:55.792] Starting Geth on Ethereum mainnet...
INFO [11-01|17:45:55.792] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [11-01|17:45:55.792] Enabling metrics collection
INFO [11-01|17:45:55.792] Enabling stand-alone metrics HTTP endpoint address=0.0.0.0:6060
INFO [11-01|17:45:55.792] Starting metrics server                  addr=http://0.0.0.0:6060/debug/metrics
INFO [11-01|17:45:55.793] Maximum peer count                       ETH=0 LES=0 total=0
INFO [11-01|17:45:55.794] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [11-01|17:45:56.547] Set global gas cap                       cap=50,000,000
INFO [11-01|17:45:56.558] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [11-01|17:45:56.558] Allocated cache and file handles         database=/data/geth/chaindata cache=2.00GiB handles=524,288
INFO [11-01|17:48:21.145] Found legacy ancient chain path          location=/data/geth/chaindata/ancient
INFO [11-01|17:48:23.265] Opened ancient database                  database=/data/geth/chaindata/ancient readonly=false
INFO [11-01|17:49:16.568] Disk storage enabled for ethash caches   dir=/data/geth/ethash count=3
INFO [11-01|17:49:16.568] Disk storage enabled for ethash DAGs     dir=/root/.ethash     count=2
INFO [11-01|17:49:18.138] Initialising Ethereum protocol           network=1 dbversion=8
INFO [11-01|17:49:18.635]
INFO [11-01|17:49:18.635] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-01|17:49:18.635] Chain ID:  1 (mainnet)
INFO [11-01|17:49:18.635] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [11-01|17:49:18.635]
INFO [11-01|17:49:18.635] Pre-Merge hard forks:
INFO [11-01|17:49:18.635]  - Homestead:                   1150000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [11-01|17:49:18.635]  - DAO Fork:                    1920000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md)
INFO [11-01|17:49:18.635]  - Tangerine Whistle (EIP 150): 2463000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [11-01|17:49:18.635]  - Spurious Dragon/1 (EIP 155): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-01|17:49:18.635]  - Spurious Dragon/2 (EIP 158): 2675000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [11-01|17:49:18.635]  - Byzantium:                   4370000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [11-01|17:49:18.635]  - Constantinople:              7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [11-01|17:49:18.635]  - Petersburg:                  7280000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [11-01|17:49:18.635]  - Istanbul:                    9069000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [11-01|17:49:18.635]  - Muir Glacier:                9200000  (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [11-01|17:49:18.635]  - Berlin:                      12244000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [11-01|17:49:18.635]  - London:                      12965000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [11-01|17:49:18.635]  - Arrow Glacier:               13773000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [11-01|17:49:18.635]  - Gray Glacier:                15050000 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [11-01|17:49:18.635]
INFO [11-01|17:49:18.635] Merge configured:
INFO [11-01|17:49:18.635]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [11-01|17:49:18.635]  - Network known to be merged: true
INFO [11-01|17:49:18.635]  - Total terminal difficulty:  58750000000000000000000
INFO [11-01|17:49:18.635]  - Merge netsplit block:       <nil>
INFO [11-01|17:49:18.635] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [11-01|17:49:18.635]
INFO [11-01|17:49:42.672] Loaded most recent local header          number=15,870,557 hash=fe10e0..9522fa td=58,750,003,716,598,352,816,469 age=20h55m7s
INFO [11-01|17:49:42.672] Loaded most recent local full block      number=15,870,557 hash=fe10e0..9522fa td=58,750,003,716,598,352,816,469 age=20h55m7s
INFO [11-01|17:49:42.672] Loaded most recent local fast block      number=15,870,557 hash=fe10e0..9522fa td=58,750,003,716,598,352,816,469 age=20h55m7s
INFO [11-01|17:49:42.672] Loaded most recent local finalized block number=15,870,479 hash=ef0b8f..72beec td=58,750,003,716,598,352,816,469 age=21h10m55s
INFO [11-01|17:49:47.839] Loaded local transaction journal         transactions=0 dropped=0
INFO [11-01|17:49:47.839] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|17:49:47.839] Chain post-merge, sync via beacon client
INFO [11-01|17:49:47.839] Gasprice oracle is ignoring threshold set threshold=2
WARN [11-01|17:49:48.116] Unclean shutdown detected                booted=2022-08-23T03:36:13+0000 age=2mo1w3d
WARN [11-01|17:49:48.116] Unclean shutdown detected                booted=2022-09-01T07:09:01+0000 age=2mo1d10h
WARN [11-01|17:49:48.116] Unclean shutdown detected                booted=2022-09-02T09:25:55+0000 age=2mo8h23m
WARN [11-01|17:49:48.116] Unclean shutdown detected                booted=2022-09-16T13:32:44+0000 age=1mo2w2d
WARN [11-01|17:49:48.116] Engine API enabled                       protocol=eth
INFO [11-01|17:49:48.228] Starting peer-to-peer node               instance=Geth/v1.11.0-unstable-468d1844-20220919/linux-amd64/go1.18.6
INFO [11-01|17:49:50.076] New local node record                    seq=1,660,867,625,814 id=05c8f483ed1db120 ip=127.0.0.1 udp=30303 tcp=30303
INFO [11-01|17:49:50.076] Started P2P networking                   self=enode://daecb556c75b8d70d7dc5c513450456517bb543704bbf81104136c5e6f8d65467478ee81508d24a666e301597decf5106a5ceea7e85161068aea907999056bc3@127.0
.0.1:30303
INFO [11-01|17:49:50.078] IPC endpoint opened                      url=/data/geth.ipc
INFO [11-01|17:49:50.078] Loaded JWT secret file                   path=/etc/keys/jwt.hex crc32=0xcbc83b4b
INFO [11-01|17:49:50.078] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors= vhosts=geth-node-mainnet-full
INFO [11-01|17:49:50.078] WebSocket enabled                        url=ws://[::]:8546
INFO [11-01|17:49:50.078] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [11-01|17:49:50.078] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
WARN [11-01|17:50:11.218] Served eth_getLogs                       conn=127.0.0.1:46290 reqid=11 duration=6.643932208s err="context canceled"
WARN [11-01|17:50:23.118] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|17:50:31.910] Served eth_getLogs                       conn=127.0.0.1:46300 reqid=4  duration=6.332393546s err="context canceled"
WARN [11-01|17:52:05.027] Served eth_getLogs                       conn=127.0.0.1:45744 reqid=12 duration=6.195762619s err="context canceled"
WARN [11-01|17:52:25.943] Served eth_getLogs                       conn=127.0.0.1:57224 reqid=4  duration=6.107904275s err="context canceled"
WARN [11-01|17:52:48.240] Served eth_getLogs                       conn=127.0.0.1:51644 reqid=6  duration=6.096511163s err="context canceled"
WARN [11-01|17:53:14.599] Served eth_getLogs                       conn=127.0.0.1:43208 reqid=5  duration=6.008904903s err="context canceled"
INFO [11-01|17:53:29.053] Etherbase automatically configured       address=0xb2E19D23405F1B16a86c0D1AD4110e59357Bd47C
WARN [11-01|17:53:33.609] Rewinding blockchain                     target=15,012,712
WARN [11-01|17:53:39.319] Served eth_getLogs                       conn=127.0.0.1:57658 reqid=6  duration=6.209518539s err="context canceled"
WARN [11-01|17:54:03.249] Served eth_getLogs                       conn=127.0.0.1:50960 reqid=9  duration=6.151890567s err="context canceled"
WARN [11-01|17:54:24.700] Served eth_getLogs                       conn=127.0.0.1:60942 reqid=5  duration=6.310364947s err="context canceled"
WARN [11-01|17:54:49.357] Served eth_getLogs                       conn=127.0.0.1:38786 reqid=12 duration=6.306824584s err="context canceled"
WARN [11-01|17:55:23.151] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:00:23.185] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:05:23.220] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:10:23.254] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:15:23.290] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:20:23.322] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:25:23.358] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:30:23.389] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:35:23.419] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:40:23.452] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:45:23.486] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|18:49:47.839] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|18:49:47.840] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|18:49:48.286] Persisted the clean trie cache           path=/data/geth/triecache elapsed=447.330ms
WARN [11-01|18:50:23.520] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|18:55:23.565] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:00:23.621] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:05:23.710] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:10:23.757] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:15:23.811] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:20:23.900] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:25:23.998] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
ERROR[11-01|19:28:24.906] Missing block in the middle, aiming genesis number=15,012,866 hash=096e67..35c886
WARN [11-01|19:28:24.908] Force rewinding till ancient limit       head=0
WARN [11-01|19:30:24.049] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:35:24.125] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:40:24.175] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:45:24.261] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|19:49:47.857] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|19:49:47.858] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|19:49:48.535] Persisted the clean trie cache           path=/data/geth/triecache elapsed=677.190ms
WARN [11-01|19:50:24.352] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|19:55:24.471] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:00:24.556] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:05:24.620] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:10:24.729] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:15:24.870] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:20:24.953] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:25:25.026] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:30:25.093] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:35:25.198] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:40:25.363] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:45:25.434] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|20:49:47.839] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|20:49:47.860] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|20:49:48.478] Persisted the clean trie cache           path=/data/geth/triecache elapsed=629.699ms
WARN [11-01|20:50:25.543] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|20:55:25.622] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:00:25.697] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:05:25.838] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:10:25.971] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:15:26.113] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:20:26.197] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:25:26.325] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:30:26.492] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:35:26.630] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:40:26.730] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:45:26.907] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|21:49:47.843] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|21:49:47.844] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|21:49:48.471] Persisted the clean trie cache           path=/data/geth/triecache elapsed=627.806ms
WARN [11-01|21:50:27.008] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|21:55:27.179] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:00:27.300] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:05:27.418] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:10:27.518] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:15:27.583] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:20:27.686] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:25:27.859] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:30:28.085] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:35:28.249] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:40:28.452] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:45:28.572] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|22:49:47.841] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|22:49:47.841] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|22:49:48.619] Persisted the clean trie cache           path=/data/geth/triecache elapsed=776.983ms
WARN [11-01|22:50:28.785] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|22:55:28.918] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:00:29.101] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:05:29.236] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:10:29.393] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:15:29.619] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:20:29.829] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:25:30.004] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:30:30.153] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:35:30.339] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:40:30.570] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:45:30.752] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-01|23:49:47.851] Writing clean trie cache to disk         path=/data/geth/triecache threads=1
INFO [11-01|23:49:47.851] Regenerated local transaction journal    transactions=0 accounts=0
INFO [11-01|23:49:48.662] Persisted the clean trie cache           path=/data/geth/triecache elapsed=810.500ms
WARN [11-01|23:50:30.926] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-01|23:55:31.157] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:00:31.433] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:05:31.654] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:10:31.904] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:15:32.076] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:16:04.879] SetHead invalidated safe block
INFO [11-02|00:16:04.997] Loaded most recent local header          number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=53y7mo1w
INFO [11-02|00:16:04.997] Loaded most recent local full block      number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=53y7mo1w
INFO [11-02|00:16:04.997] Loaded most recent local fast block      number=0          hash=d4e567..cb8fa3 td=17,179,869,184                 age=53y7mo1w
WARN [11-02|00:20:32.116] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:25:32.151] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
WARN [11-02|00:30:32.188] Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!
INFO [11-02|00:32:48.505] Got interrupt, shutting down...
INFO [11-02|00:32:48.507] HTTP server stopped                      endpoint=[::]:8545
INFO [11-02|00:32:48.508] HTTP server stopped                      endpoint=[::]:8546
INFO [11-02|00:32:48.509] HTTP server stopped                      endpoint=127.0.0.1:8551
INFO [11-02|00:32:48.510] IPC endpoint closed                      url=/data/geth.ipc
INFO [11-02|00:32:48.511] Ethereum protocol stopped
INFO [11-02|00:32:48.511] Transaction pool stopped
ERROR[11-02|00:32:48.512] Failed to journal state snapshot         err="snapshot [0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544] missing"
INFO [11-02|00:32:48.513] Writing clean trie cache to disk         path=/data/geth/triecache threads=8
INFO [11-02|00:32:48.610] Persisted the clean trie cache           path=/data/geth/triecache elapsed=97.372ms
INFO [11-02|00:32:48.611] Blockchain stopped

[rjl493456442]

Sorry for it, it's a logic in Geth that if some blocks are missing in between then we can't simply find the parent block and thus interrupt the rewinding. It's expected behavior.

We should investigate why the block bodies will be missing in the first place.

[caseybrown89]

thanks @rjl493456442!

Sorry for it, it's a logic in Geth that if some blocks are missing in between then we can't simply find the parent block and thus interrupt the rewinding. It's expected behavior.

I understand this is the way it was written. I don't understand why it needs to be this way though, if there is a last known "good" block, or even Geth could search for that itself. If at the end of the day the data can no longer be trusted, each block could probably be verified which would probably be faster than resyncing? Maybe something for the future.

We should investigate why the block bodies will be missing in the first place.

If I can provide anything here, please let me know.

Given that ~1.6TB of space was still persisted after the going back to 0, does that mean the resync is using some local data and is repopulating faster? Or it will all be overwritten as data comes in from peers?

[fjl]

Also possibly related: #22374

[MariusVanDerWijden]

Damn, sorry for dropping the ball on this. I hope you managed to full sync your node in the end. I'll close this issue for now. Please open a new one if this issue still persists