What is the upper bound of "imported new state entries"? When will it end? #15616
Comments
|
I don't exactly know "how long" it would take, but it's suggested to let it run for as long as it takes. It took me 12-15 hours on my 8-core i7 machine with |
|
@asgs I have read issues before, they don't make any sense. I just want to know how long I have to wait, and should I stop it. yesterday eth.syncing return false, and few minutes later, it print like "{ |
|
@idotial how long have u been waiting and do u have an ssd/fast internet? No u probably shouldnt stop it, but you can try... it took me like 6 hours on importing the states part yesterday. also how much ram do u have? maybe thats an issue |
|
i have the same problem i am using here is the The problem is i dont have enough disk space or i have 7GB and im not sure weather to throw away my old How much space did you need to finish the sync and did it work? |
|
Did it ever finish? |
|
@xstalpha It's still running, but it start to Imported new chain segment, so I guest it's alright. |
|
@keznerve just stop it. you can add an Portable Hard Disk, and rerun it. |
|
@ugotjelly I stop it, and restart it few times. and now the log is 'Imported new chain segment', so maybe it's will finish in several days. |
|
Hi yes it finished ...it went up to 38GB on the 19th ofDec it got me to clear out all the crap on my Mac, to save space ; the |
|
@keznerve congratulation! It seems you have better network |
|
In my case millions of |
|
I was also seeing this with current master (1.8.0). 1.7.3 is pulling in chain state properly however is too slow to catch up on non-SSD disks. I think this PR is going to be a big win. |
|
I just finished fast syncing with Geth 1.7.3 64-bit for Windows using an SSD and wanted to leave this data point for others. It took a little more than 2 days. Yep, finished syncing just short of block 5,000,000. That's about 81 million known states for 5 million blocks. The database size on disk is 66,323,816,699. I kept getting crashes using the 1.7.2 that came with the wallet even going to an SSD. Parity in warp mode was also crashing. |
|
thanks much |
|
81 million known states for 5 million blocks 😱 😨 Currently at currentBlock=5071336 and knownStates=66991785 At least I know there's some way to go yet... Been at this for ~8 hours on an Amazon EC2 m5.4xlarge. |
|
@hynese did it finish? |
|
Haven't had a chance to check yet. I was up over 80 million state at midnight last night (15 hours ago) and was still at eth.blockNumber=0 then. |
|
Yes, I'm in sync now. Finally... Several hours to go from 0 to ~85 million |
|
Appreciate you guys letting us know how many ironically named "known states" there are :-D |
|
@martindye it finished for (took 15 hours). Unfortunately after download that metrics are hidden, but i think last number of states i saw was 100M |
|
On a machine with slow disk subsystem and the new trie cache (merged in v1.8.0) it took six days to perform fast sync. Before the new cache has been implemented, the node was not able to stay in sync. |
|
is still being updated... |
|
i downloaded the Geth 1.8.1 on ethereum wallet and got locked out...unfortunately i didnt have any luck after December running fast client. I have had to ditch the Node for now. Hope it all works out soon . |
|
INFO [04-17|10:02:46] Imported new state entries count=0 elapsed=729.752ms processed=121292080 pending=2995 retry=16 duplicate=26888 unexpected=32477 |
|
approximately 130 millions, three weeks ago(state entries) |
|
Just finished today, the last knownStates I saw is |
|
As for 8th May 2018, with geth |
|
Just wanted to say thanks to the last 3 responders for actually giving a number! Every other thread on this topic just tells people to run in light mode, use MEW, etc, etc. At least I know what I'm in for here (I'm chugging along at 129mm states at the moment, hoping the target is about 140mm now). I'll endeavor to post the most recent state number when complete. Would be nice if this number could be found somewhere (like etherscan). The answer: As of May 18, it finally completed at 140,936,000 |
|
just want to know say for an instance of 8Gb of memory, what the cache setting should be? 4096? so when it's pulling states (instead of blocks), it almost seems like the current geth (1.8.8 stable) is not trying very hard to complete the job... process: Update on 5/20/2018: Update: |
I'll second that. I am also finding it hard to tell if there is an issue syncing without knowing whether I am catching up or falling behind. And a generic what to do about it. |
|
It took me approximately 36 hours to sync. 970 EVO NVMe M.2 SSD 1TB Good luck ! |
|
Sharing data of my fully synced node also, maybe it is useful to others. D:\Program Files\Geth>geth --datadir s:\Ethereum inspect
INFO [11-11|19:19:15.793] Maximum peer count ETH=50 LES=0 total=50
INFO [11-11|19:19:16.156] Set global gas cap cap=25000000
INFO [11-11|19:19:16.160] Allocated cache and file handles database=s:\Ethereum\geth\chaindata cache=512.00MiB handles=8192
INFO [11-11|19:19:27.022] Opened ancient database database=s:\Ethereum\geth\chaindata\ancient
INFO [11-11|19:19:27.046] Disk storage enabled for ethash caches dir=s:\Ethereum\geth\ethash count=3
INFO [11-11|19:19:27.051] Disk storage enabled for ethash DAGs dir=C:\Users\giuse\AppData\Local\Ethash count=2
INFO [11-11|19:19:27.059] Loaded most recent local header number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.069] Loaded most recent local full block number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.079] Loaded most recent local fast block number=11237906 hash="d8c4b8…c3f1e9" td=18641055599883751407245 age=53s
INFO [11-11|19:19:27.087] Loaded last fast-sync pivot marker number=10760885
INFO [11-11|19:19:35.100] Inspecting database count=4996000 elapsed=8.002s
INFO [11-11|19:19:43.108] Inspecting database count=10062000 elapsed=16.010s
INFO [11-11|19:19:51.115] Inspecting database count=14730000 elapsed=24.017s
INFO [11-11|19:19:59.123] Inspecting database count=19415000 elapsed=32.025s
...
INFO [11-11|19:50:45.078] Inspecting database count=1815272000 elapsed=31m17.980s
INFO [11-11|19:50:53.083] Inspecting database count=1821919000 elapsed=31m25.985s
INFO [11-11|19:51:01.090] Inspecting database count=1828550000 elapsed=31m33.992s
INFO [11-11|19:51:09.097] Inspecting database count=1835247000 elapsed=31m41.999s
+-----------------+--------------------+------------+-----------+
| DATABASE | CATEGORY | SIZE | ITEMS |
+-----------------+--------------------+------------+-----------+
| Key-Value store | Headers | 54.97 MiB | 100438 |
| Key-Value store | Bodies | 3.52 GiB | 98390 |
| Key-Value store | Receipt lists | 4.48 GiB | 98390 |
| Key-Value store | Difficulties | 7.11 MiB | 113813 |
| Key-Value store | Block number->hash | 6.04 MiB | 113813 |
| Key-Value store | Block hash->number | 439.41 MiB | 11237928 |
| Key-Value store | Transaction index | 30.13 GiB | 898783141 |
| Key-Value store | Bloombit index | 1.76 GiB | 5617664 |
| Key-Value store | Contract codes | 264.01 MiB | 44710 |
| Key-Value store | Trie nodes | 130.27 GiB | 880287080 |
| Key-Value store | Trie preimages | 2.98 GiB | 45170757 |
| Key-Value store | Account snapshot | 0.00 B | 0 |
| Key-Value store | Storage snapshot | 0.00 B | 0 |
| Key-Value store | Clique snapshots | 0.00 B | 0 |
| Key-Value store | Singleton metadata | 151.00 B | 5 |
| Ancient store | Headers | 4.75 GiB | 11147907 |
| Ancient store | Bodies | 108.13 GiB | 11147907 |
| Ancient store | Receipt lists | 50.63 GiB | 11147907 |
| Ancient store | Difficulties | 173.74 MiB | 11147907 |
| Ancient store | Block number->hash | 404.00 MiB | 11147907 |
| Light client | CHT trie nodes | 0.00 B | 0 |
| Light client | Bloom trie nodes | 0.00 B | 0 |
+-----------------+--------------------+------------+-----------+
| TOTAL | 337.97 GIB | |
+-----------------+--------------------+------------+-----------+
ERROR[11-11|19:51:16.769] Database contains unaccounted data size=126.40KiB count=2748 |
|
I'm trying to do a fastsync on a local server but it seems the disk speed drops off after about 30 minutes of "importing new state entry". I'm running the sync process on machine with SSD, 16GB RAM and i5. Cache is setup to 12GB (fills up only to 50%). The process starts with about 100-150MB/s I/O and after 30 minutes drops to 10-20MB/s I/O. Is this normal?
|
|
I just finished syncing a new node the final count was |
|
just hit |
|
Finished syncing last night, final was: Hardware: |
|
it would help if you write your hardware too not just the number... |
|
I just synced another eth1 full node in 64 hours, processing 650.634.632 state entries, using 282.75 GIB. Full stats below. Please note that during synchronization the same machine was running a Teku beacon node, so performance could have been better without it. I wanted to test if those two clients can both run smoothly on this mini PC and, spoiler alert, yes they can 😉.
|
|
What I am looking for from this issue is an approximate formula for states, and some link to a blog or something that has accumulated a bunch of failures of when your system is not ever going to catch up to the header block. The current method I am using uses the estimate of 3-6k state changes per block (Reference) . Of those 3-6k changes, 2-4k are additions to the state tree and 1-2k are deletions. So delta ~ 3-6k, net ~1-2k per block. If we use these estimates on @Neurone local state above to make a lower bound ( 650,634,632 in 64 hours) "Total states" = 650,634,632 - (3000x4x60x64) = 604554632 at the start of sync, and "Total states" = 604554632+(1000x4x60x64) = 619914632 at the end of sync. I think this is a good formula for estimating in lieu of a better way -- so far for me it has correctly predicted my inability to catch up to the header block, I just can't quite figure out my system's issue (my data here). The 3-6k seems outdated so there is also that. Any formulas or indicators for "never catching up"?
|
|
Running a 4gb Raspberry Pi and approaching 900 million state entries and 325gb used.
|
|
FYI, I finished syncing last night: State entry: 664 401 205 Hardware NUC i5-8259U -16 GB RAM - 1TB NMVE Sabrent Q |
|
Just an update for February: 11797042 blocks Node synced in 12 hours on a i7, 32Gb RAM and 500GBSSD in RAID1, dedicated server. |
|
Not sync yet with 753780733 states :( |
|
Hey, today I finally managed to synchronize the node, I leave you the latest data in case it is useful for someone, it has taken a lot, practically a month :)
|
|
Re-Sync, very close to sync complete
|
|
Currently Command SSD used, 8G RAM, currently about 200k state entries per hour, it will take over a month to arrive 900m! Is there anything that I can do? |
|
I am currently at 1059769034 states and still syncing on a Pi 4. Starting to think it may never catch up. |
|
Upped the clock rate, still not caught up.
|
|
wow 1059769034, I am only at 810953009, still got a long ways to go :/ about two months ago it took me about three days to sync, now its taken a week already, and its super slow, I think it was related to using the lastest client before the berlin fork as soon as it went live the state sync went up significantly but it has slowed down again :/ |
|
also about three months ago the state count was ~700000000 seems like it goes up fast. |
|
OK I discovered that state entries are misleading, I copied the chain over from my synchronized machine and it started at a smaller number. I believe they count a certain number from where it started to where it is now. The longer it takes the higher that number can go, so I believe tracking them here is pointless. |
|
It's been a month and knowns states are increasing. should i stop my server and forget about sync? because known states numbers keep changing from 40K to 50K to 35K and next hour it goes back to 40K. Your help will be appreciated. This is what I see
Best Regards, |
|
Don't worry, I also thought about turning off many times, but you just have to let it work, it is normal that it takes a long time, do not despair :) |
|
I just finished a fast-sync in the past couple days, unfortunately I had left it running whilst being away so I can't confirm the final number but I saw up to 1,000,000,000 and estimate that at time of writing it's closer to 1,100,000,000. For those in the thread commenting about Raspberry Pi's I completed the sync with the following setup: Pi 4 8gb
Unfortunately I tried several configfurations and didn't keep perfect records on the speed increases however: based on the final speeds achieved with the above configuration, I estimate a Pi 4 8gb could sync in about 4-6 days as of July 2021. EDIT: -cache size 2048, -max_peers 40. Running as a systemd service. |
|
Fast-sync is no longer available in geth, so I'm closing this |
and others
System information
Geth version:
1.7.2OS & Version: OSX
Expected behaviour
geth --fast should finish soon.
Actual behaviour
it run for 3days and print “Imported new state entries count=384 elapsed=26.970ms processed=50023987 pending=33074 retry=0 duplicate=19087 unexpected=47765” constantly
Steps to reproduce the behaviour
run geth --fast in console
Backtrace
INFO [12-06|07:08:00] Imported new state entries count=1259 elapsed=12.971ms processed=50014526 pending=34891 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:08:23] Imported new state entries count=774 elapsed=8.950ms processed=50015300 pending=34311 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:08:31] Imported new state entries count=1125 elapsed=9.513ms processed=50016425 pending=33428 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:08:39] Imported new state entries count=1061 elapsed=11.198ms processed=50017486 pending=32566 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:08:49] Imported new state entries count=1314 elapsed=12.041ms processed=50018800 pending=31248 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:09:10] Imported new state entries count=1028 elapsed=10.446ms processed=50019828 pending=30496 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:09:25] Imported new state entries count=1241 elapsed=10.423ms processed=50021069 pending=30088 retry=0 duplicate=19087 unexpected=47765
INFO [12-06|07:09:37] Imported new state entries count=777 elapsed=6.224ms processed=50021846 pending=29851 retry=26 duplicate=19087 unexpected=47765
The text was updated successfully, but these errors were encountered: