state.(*cachingDB).CopyTrie encounter panic: unknown trie type <nil>

[darren]

System information

Geth
Version: 1.11.0-unstable
Git Commit: d10c280
Git Commit Date: 20220831
Architecture: amd64
Go Version: go1.19
Operating System: linux

Expected behaviour

should not panic

Actual behaviour

panics

Steps to reproduce the behaviour

1. geth --cache 8000 --http --http.api eth,net,engine,admin --authrpc.jwtsecret /tmp/jwts
2. prysm.sh beacon-chain --http-web3provider=http://localhost:8551 --mainnet --jwt-secret=/tmp/jwts

After this panic, restart the node can not sync any more due to

##############################
 
ERROR[09-01|09:31:15.483] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [09-01|09:31:15.483] Synchronisation failed, dropping peer    peer=774017fcc3478b6527634b31b07b524a36f8e8a17afa9658260001f7622ce623 err="retrieved hash chain is invalid: invalid merkle root (remote: 7cc7e88cb15e7431fd61cdc8742decc9d1909dde5ff7e3e50a96006e07e1b954 local: 9168df37cb64cde9b78d788b0aa3973f6330a42c0a962de64ec2d4f9a8968f43)"

Backtrace

INFO [08-31|23:09:56.106] Writing clean trie cache to disk         path=/home/darren/.ethereum/geth/triecache threads=1
INFO [08-31|23:09:56.108] Regenerated local transaction journal    transactions=0 accounts=0
INFO [08-31|23:09:57.822] Persisted the clean trie cache           path=/home/darren/.ethereum/geth/triecache elapsed=1.715s
INFO [08-31|23:10:01.500] Imported new chain segment               blocks=1  txs=142  mgas=9.783   elapsed=126.382ms   mgasps=77.406  number=15,447,306 hash=dd5b11..f60c45 dirty=1.20GiB
INFO [08-31|23:10:01.529] Unindexed transactions                   blocks=1  txs=119  tail=13,097,307 elapsed=27.225ms
INFO [08-31|23:10:05.960] Imported new chain segment               blocks=1  txs=199  mgas=16.917  elapsed=172.884ms   mgasps=97.854  number=15,447,307 hash=7e8bdd..172e5b dirty=1.20GiB
INFO [08-31|23:10:05.963] Unindexed transactions                   blocks=1  txs=98   tail=13,097,308 elapsed=2.536ms
INFO [08-31|23:10:22.154] Imported new chain segment               blocks=1  txs=126  mgas=12.818  elapsed=106.381ms   mgasps=120.487 number=15,447,308 hash=a2c716..92b181 dirty=1.20GiB
INFO [08-31|23:10:22.156] Unindexed transactions                   blocks=1  txs=82   tail=13,097,309 elapsed=1.620ms
WARN [08-31|23:10:48.614] Trie prefetcher failed opening trie      root=f5a1d3..658db4 err="missing trie node f5a1d3d1a4a78d9ebc35976f56fa6eb998e509062018fdb48cd4968e87658db4 (owner 4921d0f888dd702cdb6067f0b62c93322db732d50df0ccbcfba9efe8a9a84983) (path ) <nil>"
WARN [08-31|23:10:48.614] Trie prefetcher failed opening trie      root=ed62f7..652246 err="missing trie node ed62f7aefd169cd03ff7a3ffd3c76ab64827fdb78d34f3e436fd39e5a2652246 (owner 05411a03c890840e8be0034e02cb7a1a877583e9b4d3eebf6cc4725948f48b5c) (path ) <nil>"
INFO [08-31|23:11:02.073] Imported new chain segment               blocks=1  txs=339  mgas=30.020  elapsed=193.029ms   mgasps=155.520 number=15,447,309 hash=40e021..a72094 dirty=1.20GiB
INFO [08-31|23:11:02.074] Unindexed transactions                   blocks=1  txs=48   tail=13,097,310 elapsed=1.171ms
WARN [08-31|23:11:02.177] Trie prefetcher failed opening trie      root=ed62f7..652246 err="missing trie node ed62f7aefd169cd03ff7a3ffd3c76ab64827fdb78d34f3e436fd39e5a2652246 (owner 05411a03c890840e8be0034e02cb7a1a877583e9b4d3eebf6cc4725948f48b5c) (path ) <nil>"
WARN [08-31|23:11:02.177] Trie prefetcher failed opening trie      root=f5a1d3..658db4 err="missing trie node f5a1d3d1a4a78d9ebc35976f56fa6eb998e509062018fdb48cd4968e87658db4 (owner 4921d0f888dd702cdb6067f0b62c93322db732d50df0ccbcfba9efe8a9a84983) (path ) <nil>"
panic: unknown trie type <nil>

goroutine 170 [running]:
github.com/ethereum/go-ethereum/core/state.(*cachingDB).CopyTrie(0xc0df0e2f38?, {0x0?, 0x0?})
        github.com/ethereum/go-ethereum/core/state/database.go:168 +0x153
github.com/ethereum/go-ethereum/core/state.(*triePrefetcher).copy(0xc14bf135f0)
        github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go:129 +0x3c9
github.com/ethereum/go-ethereum/core/state.(*StateDB).Copy(0xc0fec5bd40)
        github.com/ethereum/go-ethereum/core/state/statedb.go:714 +0xeb2
github.com/ethereum/go-ethereum/miner.(*worker).updateSnapshot(0xc0096a18c0, 0xc1adc40a00)
        github.com/ethereum/go-ethereum/miner/worker.go:822 +0x3b9
github.com/ethereum/go-ethereum/miner.(*worker).commit(0xc1e39f4dc0?, 0x0?, 0x17dbb00?, 0x0?, {0x8?, 0xffffffffffffffff?, 0x2248500?})
        github.com/ethereum/go-ethereum/miner/worker.go:1165 +0xf9
github.com/ethereum/go-ethereum/miner.(*worker).commitWork(0xc0096a18c0, 0xc0df0e3b7c?, 0x0, 0x630f7a06)
        github.com/ethereum/go-ethereum/miner/worker.go:1123 +0x279
github.com/ethereum/go-ethereum/miner.(*worker).mainLoop(0xc0096a18c0)
        github.com/ethereum/go-ethereum/miner/worker.go:534 +0x85e
created by github.com/ethereum/go-ethereum/miner.newWorker
        github.com/ethereum/go-ethereum/miner/worker.go:293 +0x776

[rjl493456442]

You ran the 1.10.22 previously right? You have to set the head back to the point you upgraded to 1.10.22 and re-import the chain in between. We had a regression in 1.10.22 which will lose some data.

[darren]

@rjl493456442

It seems that you have fixed the original panic in #25575, so I am going to close this.

You ran the 1.10.22 previously right?

No I am running on the dev branch, not the stable channel. tried debug.setHead serveral times, still got

ERROR[09-02|14:38:42.796] Failed to retrieve beacon bounds for bad block reporting err="beacon sync not yet started"
WARN [09-02|14:38:42.797] Synchronisation failed, dropping peer    peer=d5c94cfcfe2358133274d9a3c6d9e4bc766f0c1355c05565185e89ecb1e7f6c2 err="retrieved hash chain is invalid: invalid merkle root (remote: 7cc7e88cb15e7431fd61cdc8742decc9d1909dde5ff7e3e50a96006e07e1b954 local: 9168df37cb64cde9b78d788b0aa3973f6330a42c0a962de64ec2d4f9a8968f43)"
WARN [09-02|14:38:42.797] Synchronisation failed, retrying         err="peer is unknown or unhealthy"
INFO [09-02|14:40:14.216] New local node record                    seq=1,655,740,187,340 id=2210dd52ea14560d ip=123.122.163.59  udp=30303 tcp=30303

Geth
Version: 1.11.0-unstable
Git Commit: 511bf8f18801520bf4e0c7e4d098a17d0665bc89
Git Commit Date: 20220901
Architecture: amd64
Go Version: go1.19
Operating System: linux
GOPATH=
GOROOT=