Replies: 89 comments 75 replies
-
|
What out of sync timeframe you are talking about? I am able to catch up after beeing out of sync for 6 hours easily with an old i5 processor and an low budget SSD |
Beta Was this translation helpful? Give feedback.
-
|
To me it looks it depends on your "luck" and peers you are connected to before you reached limit of peers. |
Beta Was this translation helpful? Give feedback.
-
|
this is example from farmer node (no harvesters) 47 seconds per block. |
Beta Was this translation helpful? Give feedback.
-
|
Ja, that sound legit. Take a look into #3298. I am currently trying to hard-ban peers that I can get no connection to so that the garbage collector can remove them from ChiaServer's internal peer-list |
Beta Was this translation helpful? Give feedback.
-
|
Also, chia_full_node is bounded by CPU, it shows 100% CPU usage, while in this terrible sync. |
Beta Was this translation helpful? Give feedback.
-
|
Another thing I see from logs, I am feeding known blocks to other nodes with speed of light, but this also, probably, one of reasons at CPU bounding. |
Beta Was this translation helpful? Give feedback.
-
|
I echo this same problem - got out of sync a few days ago, and unable to regain sync. After restart multiple times and complete db delete, in 24 hours, I am only a little over half-sync'd. |
Beta Was this translation helpful? Give feedback.
-
|
One more thing, looks like sometimes it just getting stuck on particular block, I presume, waiting something from particular peer and not getting |
Beta Was this translation helpful? Give feedback.
-
at this moment this solution work... I think this only peers problem. If i have luck, sync work great, bad peer and not synced for 6hours. Synchronization is broken, totally. This isn`t problem of db, cpu or ssd, network. Only app have big problem. Manually wallet restart can helps but it afffect farming... |
Beta Was this translation helpful? Give feedback.
-
|
Can you point towards procedure? |
Beta Was this translation helpful? Give feedback.
-
|
i work on it... this is sample :) App looks like use only first added peer, but that is bad solution for farming. |
Beta Was this translation helpful? Give feedback.
-
What speed of sync you have in this case? number of seconds per block |
Beta Was this translation helpful? Give feedback.
-
|
40s per block as long as i`m lucky Only wallet restart helps for me. |
Beta Was this translation helpful? Give feedback.
-
it is still slower than chain grows. |
Beta Was this translation helpful? Give feedback.
-
|
Protocol and software are terminally ill. |
Beta Was this translation helpful? Give feedback.
-
|
I notice that my old server running chia is just fine, but any machine that I try to install and run from scratch, e.g. not actually copy all files from .chia/main-net, then it never completes. My old original from months ago is fine, but on other machines say ubuntu 18, or 20, I have tried to build new fresh install blockchains, and its takes 2-3 days to get up to current block, but no sync ever happens, My original system on linux mint, is still ok, so long as I run, but I have found that I can't mix plotting with the blockchain as the server interferes with my cmd-line linux plotting, so I moved all the blockchain to other machines, to which they never sync .. so what I don't care chia doesn't payout anyway, just keep plotting and using hpool to earn XCH |
Beta Was this translation helpful? Give feedback.
-
|
so, that's a big F.. Something is VERY wrong..... |
Beta Was this translation helpful? Give feedback.
-
this is the best I could get :) |
Beta Was this translation helpful? Give feedback.
-
|
If those are the only peers you have, I would suggest opening port 8444. My node is currently connected to 75 peers and only one of those peers is at height 0. |
Beta Was this translation helpful? Give feedback.
-
|
If I had 8444 closed, I'd be unable to sync :) Those nodes eventually updates to nonzero block, but then getting stuck. And as I mentioned earlier, I was able to get up to 300 peers. Quantity doesn't matter. Then only short periods I have it running in sync, is when I get non-eu peers for some time (usually 15-30 min), and then they disconnect and I go back to long syncing |
Beta Was this translation helpful? Give feedback.
-
|
which property is for the max wait time? |
Beta Was this translation helpful? Give feedback.
-
|
Have no clue. Didn't find. |
Beta Was this translation helpful? Give feedback.
-
|
Update. |
Beta Was this translation helpful? Give feedback.
-
|
So I am having problems syncing and it appears it may be related to my CPU utilization rate? so much for a "green" crypto...it's a resource hog.... |
Beta Was this translation helpful? Give feedback.
-
|
To me it looks like this is the case. |
Beta Was this translation helpful? Give feedback.
-
|
I am currently plotting and my CPU is staying around 98% all the while I am not synced...so now I need a separate machine for the node and the plotter it seems |
Beta Was this translation helpful? Give feedback.
-
|
but then you also load your CPU with plotting. |
Beta Was this translation helpful? Give feedback.
-
|
separate machine will no necessary solve :) As you still need remote peers to be able to communicate. |
Beta Was this translation helpful? Give feedback.
-
i thought me not being able to sync was related to my plotting using all my CPU resources...? |
Beta Was this translation helpful? Give feedback.
-
|
Try killing the wallet and letting everything else run. Not sure why but when I killed my wallet process it started syncing normally and hasn't dropped out. Been running about 4 hours like that where I was having major syncing issues before. The exact steps I followed were:
The reason I believe the wallet may have had a hand in the issues is because I've tried all the other steps multiple times already and they would only let me catch up to a few blocks and maybe stay synced for 30 mins but it does line up with some of the stuff you mentioned with limiting the outgoing connections. |
Beta Was this translation helpful? Give feedback.
-
|
just to be clear don't uninstall, try the stop wallet-only thing first and see where that gets you. |
Beta Was this translation helpful? Give feedback.
-
|
The wallet GUI is broken I don't even try to run it anymore for anything, I just run cmd-line for everything, and on plotting machines, I have no blockchain running, and never use this 'plotman' it likes to rewrite all the yaml files and create a mess from hell. Just keep it simple. Best solution if your new is fall back to 13 for fresh install and get that working for plotting too, post 13, 14, 15 it just got worse, I think its like the pool deal where they promised may 17 and promised main-net may 5, these people have made too many promises, and now they're just sending out broken software. Needs to be a moratorium on everything, until something is working, Now that chia is rich, why not hire an inhouse test team? Why is CEO hoffman still fielding 101 bs on github? Why is Bram still making videos about the IPO? There's some good people here, hell richard kiss has been around forever on the python pycoin, knows his stuff; The problem is the code for main-net was NOT ready, and HOFFMAN just deletes any post that tells the truth. |
Beta Was this translation helpful? Give feedback.
-
|
ver: 1.1.643: 24h no sync problems |
Beta Was this translation helpful? Give feedback.
-
|
I too am having this issue, not sure why it was moved to discussion, it is clearly a bug. I am on Ubuntu, yesterday at about 4pm GMT this issue just came out of nowhere, nothing changed, I did stop the node or anything, it was running perfectly one minute, then suddenly dropped out of sync and since I have not been able to get it back. I too am seeing the 0 sized nodes and sending data to lots of nodes but only seeing a small number sending to me. I lag about an hour or two behind usually It is using a lot of CPU power while it is in this state, generally 50-90% where when it was synced it would use about 20% at most. |
Beta Was this translation helpful? Give feedback.
-
|
0 sized nodes is the signs that nodes can not even deliver to you their block height. And yes, as I wrote yesterday, it will hit more and more people. once you got little delay and got out of sync and lost at least couple of "good peers", there is no really way back :) You can have accidental "in-sync" moments and farming for half an hour, but it is gone really :) |
Beta Was this translation helpful? Give feedback.
-
|
this is exactly what i am seeing, occasionally I get in sync but only for 5 minutes tops then it drops off the cliff again, there seem to be more and more people having this issue. I am starting to wonder if its an attack, or maybe the client or server side code is not very optimised and lets hope the next release fixes this. I am close to giving up on this project if I'm being hones, I know many people already have for this reason, and many more about to. |
Beta Was this translation helpful? Give feedback.
-
|
try killing the wallet through the CLI. It's not a long term fix but it'll allow you to make use of your plots until a fix is implemented. It should be: chia stop wallet-only |
Beta Was this translation helpful? Give feedback.
-
|
tried that numerous times, doesnt really help |
Beta Was this translation helpful? Give feedback.
-
This is not new, been running for months, and nothing ever stays sync'd once its sync, even in the best of times it drops out 10 minutes every hour and don't get me started on install-gui.sh, unbuntu is the default and it doesn't work, on all versions I must issue "npm install electron", the .sh files are worthless, too generic, because most of the devs are on macintosh, not connected to real world. |
Beta Was this translation helpful? Give feedback.
-
|
if we keep this thread active then maybe it will get the focus of the developers, this is a problem that so many people are suffering from, they cannot ignore this surely |
Beta Was this translation helpful? Give feedback.
-
|
I am not sure :) |
Beta Was this translation helpful? Give feedback.
-
|
its such a pain in the ass TBH, its not as if i was even doing anything when it suddenly started happening, its not ports, they are forwarding perfectly fine, I have even rebooted my router to see if it was that. I have about 30 connections to my node, only most of them have a block size of 0 and the rest are less synced than I am so there is no way I can ever catch up. I am really considering abandoning this project, the dev team moved this thread from bug to discussion without even understanding the issue and how far reaching it is now. I get that most of the time when this happens it is due to port forwarding issues, however this time it is not, many people are having this issue suddenly. |
Beta Was this translation helpful? Give feedback.
-
|
A way we can mitigate this: you can advertise your IP address here if you have the port open, and others can add you to their peers manually. That way you will have up-to-date connections and you can catch up. |
Beta Was this translation helpful? Give feedback.
-
|
LC3okIPQ8iNHlNvMixChd0KKOfJxiaoV59heeWHcB89DvQbaj3lfCs4UMW8dRRfcDo5KbLJbOQSc7+JqoEA2mO57Qra0MPDrnf75+/Cmjz2Pv3SbhWMQzaZeXL7ZnH0Kz7WOgD/meHR5hl+lblih4nVpX3ogsyOxpTC+9+nX5PU= hopefully that works lol |
Beta Was this translation helpful? Give feedback.
-
|
I got the address and it performs a handshake, requests your peers and then closes the connection. how many peers do you have at the moment? |
Beta Was this translation helpful? Give feedback.
-
|
lol 72 :) i have just reinstalled so it has everyone and their dog connected |
Beta Was this translation helpful? Give feedback.
-
|
Guys, don't waste time. Unless you have a way to make this connection permanent and ensure that you peers aren't getting overloaded and can talk to each other obstruction free. |
Beta Was this translation helpful? Give feedback.
-
|
i managed to connect. Let's see where this goes. Riz, don't forget to stop the wallet process, might also want to disconnect some peers. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know if it's simply anecdotal, but the wallet delete comment from jcampos624 got me thinking. My node has been stable since I got 0.0000000001 XCH in my balance from faucet.chia.net To be fair, I also deleted peer_table_node.sqlite when I did a restart, so that might also have been the fix. Only been up 36hrs so I wouldn't call that 'stable', but it is the longest I have been up for. |
Beta Was this translation helpful? Give feedback.
-
|
my best peer vs. introducer |
Beta Was this translation helpful? Give feedback.
-
|
I also tried deleting peer_table_node.sqlite after reading in keybase support channel that it may be necessary if full node public IP changes during syncing. Did not spot a difference. Now after reading jcampos624 comment I stopped wallet and I'm seeing some positive results. Too early to say if I will get sync though. |
Beta Was this translation helpful? Give feedback.
-
|
Several hours fully synced now, after "chia stop wallet-only" . Thanks jcampos624 ! |
Beta Was this translation helpful? Give feedback.
-
|
Ok, so if the wallet isn't running, what happens if you actually score some XCH by farming? How will you receive it? |
Beta Was this translation helpful? Give feedback.
-
|
blockchain is the ledger, it basically says you won. it doesn't matter if your wallet is synced but if you want to trade you will have to sync it or find some other way to access it. |
Beta Was this translation helpful? Give feedback.
-
|
I didn't have any connection problems but to be sure I opened port 8444 and some problems started. I have 2 perrów and this is the second time I have been disconnected. Does anyone know why this is so? |
Beta Was this translation helpful? Give feedback.
-
|
I go back to 1.1.13 and now it is syncing again |
Beta Was this translation helpful? Give feedback.
-
|
As the blockchain is growing into a formidable size, would it be worthwhile or possible for a hacker to attack the syncing process for a profit leading to this sync plague? |
Beta Was this translation helpful? Give feedback.
-
Me too, I went back to 1.1.13 for plotting as well post 13, everything about chia-net is broken. Dudes you aint' got a memory problem, you got a dev problem, the only thing that I"m impressed with is chia-lisp, but it was hell it get it working, again all the install was wrong |
Beta Was this translation helpful? Give feedback.
-
I'm curious about this: if banning peers keeps this from happening, then it follows that a bad actor could create 'poison peers' Something like 50% of rewards are currently flowing to the top ~6 farmers? https://www.chiaexplorer.com/charts/topFarmers If a peer can poison another peer and keep it from synching, but for whatever reason (network topology, firewalls etc.) some peers cannot, or know not to connect to those poisoned peers, then they don't have to compete with all the other network space. Peers suffering desync then effectively represent zombie space on the network which you don't have to compete with. We could probably check this pretty easily: the allocation of rewards by space should tend towards ideal distribution for each farmer. If all peers are affected equally, then this should balance out. If some farms are consistently trending positively against ideal (not suffering random loss of sync), while others are suffering from random loss of sync, then that may suggest that either an exploit has been discovered and deployed, or there is some other factor. Would it make a difference if the majority, or a large subset of the network leaders were on a subnet to which the rest of the network periodically lost connection with - causing that group to pull ahead and effectively desync all other peers? Just thinking out loud here |
Beta Was this translation helpful? Give feedback.
-
|
still this same, and new :) 2021-05-18T18:11:48.534 full_node chia.rpc.rpc_server : WARNING Sending data failed. Exception Traceback (most recent call last): 2021-05-18T18:14:07.978 full_node chia.full_node.full_node: WARNING Invalid response for slot None During handling of the above exception, another exception occurred: Traceback (most recent call last): 2021-05-18T18:16:40.321 full_node full_node_server : WARNING Banning 183.18.47.205 for 10 seconds |
Beta Was this translation helpful? Give feedback.
-
|
I like the idea that was posted about finding the bad peers and perm ban them. I just don't know how to identify them, and how to perm ban them. |
Beta Was this translation helpful? Give feedback.
-
|
How would you classify bad peers? Maybe they are strugglers just like the rest of us. So a permanent ban maybe too harsh. Maybe temporarily ban them if they're using old Chia version. |
Beta Was this translation helpful? Give feedback.
-
|
old bitcoin used to have a seed-list you needed to have that to boot-strap the blockchain; perhaps somebody needs to put together here a rock solid list of IP's, but I'm going to guess there aren't any, like the guy here says, everybody is hurting, |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
try this.....
|
Beta Was this translation helpful? Give feedback.
-
|
I'm up with the fix above since last night.. fingers crossed it stays up. |
Beta Was this translation helpful? Give feedback.
-
|
Why are the flaps not visible on the official website? The last one is 1.1.5.
|
Beta Was this translation helpful? Give feedback.
-
|
I don't think the issue is universal. Each time I reboot / lose sync due to closing the GUI, re-syncing takes me anywhere from 1 to 5 minutes, tops. I'm in Europe. Would be interesting to investigate if it's related to CPU / network speed / network latency / luck. |
Beta Was this translation helpful? Give feedback.
-
|
Trying 1.1.643 now https://github.com/Chia-Network/chia-blockchain/actions/runs/853387813 synced up already, lets see if it holds it.... |
Beta Was this translation helpful? Give feedback.
-
|
I've been running now for a few hours on 1.1.643 completely synced with no drop outs. no other changes made other than installing 1.1.643 |
Beta Was this translation helpful? Give feedback.
-
|
The answer is 1.1.643 |
Beta Was this translation helpful? Give feedback.
-
|
I run Win10, the link provided is only an .exe |
Beta Was this translation helpful? Give feedback.
-
|
Why is there no latest version on the official chia.net website? |
Beta Was this translation helpful? Give feedback.
-
|
dunno... maybe soon, the link provided was from github |
Beta Was this translation helpful? Give feedback.
-
|
wont work on Ubuntu then unfortunately. |
Beta Was this translation helpful? Give feedback.
-
|
Is it safe? |
Beta Was this translation helpful? Give feedback.
and others
-
You can see that there is a number of sync bugs and discussions. And they are appearing with increasing rate.
I looked into it for last 2 days and outcome is simple. Once number of transactions increased, sync speed is lagging speed of grows of blockchain.
In my test which goes for last 36 hours which I started from scratch, I can't catch current block, and this is happening on hardware you can't blame: threadripper, very SSD for DB, real 1GbE connection.
This means once typical node got out of sync for whatever reasons, it is difficult to catch up. And this should be reason of this massive surge of out of sync issues.
Beta Was this translation helpful? Give feedback.
All reactions