Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node losing sync and needs manual connect to peers [BUG] #3298

Closed
nitram-ydna opened this issue May 1, 2021 · 83 comments
Closed

Node losing sync and needs manual connect to peers [BUG] #3298

nitram-ydna opened this issue May 1, 2021 · 83 comments
Labels
bug Something isn't working stale-issue flagged as stale and will be closed in 7 days if not updated

Comments

@nitram-ydna
Copy link

Describe the bug
In the GUI the node loses sync several times a day. It can only be reconnected by manual connection to peers and doesn't look up peers on its own. If a manual connect is made, it occasionally detects another peer automatically.

To Reproduce
Leave the GUI alone plotting...
When it loses sync, try to reconnect.

Expected behavior
It is expected that the node can keep sync itself

Screenshots
chia_losesync

Desktop (please complete the following information):

  • Windows
  • 10 pro
  • Ryzen 5800x
  • 32 GB ram

Additional context
Luckily I have two plotting computers. The other computer remains synced by itself and has a list of peers.
Usually, both node.chia.net and chiadb.net are occupied when I try to connect, but I can find another 8444/8444 IP-address in the list of my second computer to use for the unsynced computer. Unfortunately, it usually takes a long time to find a node that accepts connections...

This bug is a deathstroke to Chia. If people have a hard time even winning blocks, then it won't be easier if they cannot sync...

@nitram-ydna nitram-ydna added the bug Something isn't working label May 1, 2021
@SSchiaforever
Copy link

SSchiaforever commented May 1, 2021

Have the same issue. It started to happen yesterday. I also have two plotting PCs, different wallets. One is syncing good, the second keep losing peers. I can see that its trying to connect but failing.

@dragonfly-net
Copy link

same

@guo-tokenlab
Copy link

same here

@JoNiLiGhT
Copy link

Same

@HughRyu
Copy link

HughRyu commented May 1, 2021

same here

@simondarrens
Copy link

same. i've not been able to sync since 4-27.

@fitzzz
Copy link

fitzzz commented May 2, 2021

Yea the syncing is pretty bad. Sometimes I randomly check my farmer and it is not sycned and has no peers. I try to copy paste node IPs and port from my other farmer and nothing happens... pretty lame.

@tdiz
Copy link

tdiz commented May 2, 2021

same. lose one night while waiting sync.
tips to find last connected nodes: use https://chia.keva.app/ to find last added nodes and type IPs from there in "connect to other peers" GUI section with port 8444.
it is described in wiki chia at "8444 sync issues" part

@Unwrt
Copy link

Unwrt commented May 2, 2021

1.1.3 was released, and at least it solved my sync problems

@santiagovirus
Copy link

santiagovirus commented May 5, 2021

1.1.3 solved the problem, do not use 1.1.4. I confirm this.

@pi1541
Copy link

pi1541 commented May 5, 2021

Getting the same.
If you have to manually type in the IP addresses of the peers can you please auto focus the "IP address / host" input field when the "Connect to other peers" dialog opens after clicking "CONNECT TO OTHER PEERS"?

@twitchplaystrading
Copy link

Had the same problem on 1.1.3. Updated to 1.1.4 and still having the same problem.

@mrwhy-orig
Copy link

mrwhy-orig commented May 7, 2021

Having the same problem. In Germany it is common being disconnected by the ISP after 24hr. That said, after disconnect I'm facing the resync issue. I have to close the Chia App and start it again.

@KrakkenSmacken
Copy link

KrakkenSmacken commented May 7, 2021

Same.
This seems to be a pretty important problem to solve.
Like bit torrent, available peer discovery should be built in. I have connected to some peers that had abysmal sync speeds without knowing why, or that I should be actively hunting to hit higher numbers of peers. This break, before looking here and realizing the core problem, caused me to have to choose between farming, and losing plot creations in progress.

@fiveangle
Copy link
Contributor

fully portmapped system that was set with introducer-or.chia.net (my closest as per #support macrobot on keybase) lost sync today with over 52 peers. checking and all of them were below my height. this should not be possible unless i am the only top node, which is obviously not the case. this doesn't bode well for the longevity of xch unless they can more smartly design their node ranking and introducer system.

@TradingDutchNL
Copy link

TradingDutchNL commented May 8, 2021

Same here. Using 1.1.4 Suddenly there is a Full Node connecting, Now it's syncing.. But for how long?

@Cinorid
Copy link

Cinorid commented May 9, 2021

same
also, this hosts not accepting connection from my IP address and I have to use VPN for syncing.

  • North Asia introducer-apne.chia.net:8444
  • South Asia introducer-apse.chia.net:8444
  • Western North America: introducer-or.chia.net:8444
  • Eastern North America introducer-va.chia.net:8444
  • Europe: introducer-eu.chia.net:8444
  • North America: node-or.chia.net:8444
  • Europe: node-eu.chia.net:8444
  • Asia: node-apne.chia.net:8444

@nslsa661
Copy link

nslsa661 commented May 9, 2021

Same issue here. So, is this ever going to get fixed?

@SirIain
Copy link

SirIain commented May 9, 2021

I'm still on 1.1.3 and its synced for 20 mins then not for 20 mins. Driving me mad as I have to time with plotting to update and restart but not farming while messing about.

@spodee4
Copy link

spodee4 commented May 9, 2021

Agree. Same problem. Just started a day ago. Checked ports, 8444 open. Trying to reset, uninstall etc etc. done it all. Very frustrating.

@KrakkenSmacken
Copy link

Just for those here, my workaround for the problem of plotting and farming on the same device, is to personally prioritize plotting now.

When I see a de-sync, I use this site (https://chia.keva.app/) to find new IP addresses, and run through all the
introducers.

It's manual, and a pain, and doesn't always work or stay stable, BUT it prevents me from having to restart plot building.

@SavageStyle
Copy link

Please do something about sync - there is absolutely no way i can forward port 8444, my ISP prevents that. Please make outgoing connections better atleast!

@markusmazurczak
Copy link

Same here. Once a day the fullnode and the wallet go out of sync and even after waiting for 3h the full node nor the wallet got synced again.
I have to restart all.

I am running 1.1.15

@Jufik
Copy link

Jufik commented May 11, 2021

Yup, same here.
The node just won't sync, 80 useless plotted TiB waiting, quite frustrating.

Made it work adding random introducers, but this will be hurtfull for chia when average users will kick in.
chia show -a chiadb.net:8444 has helped

@nslsa661
Copy link

Well, I'm about done with this BS. I finally figured out a work around to open port 8444 on my router and it seemed that it had mitigated my constant sync\not available issues. Nope. Worked for about 4 hours, and then all 243 plots went unavailable again. So I tried adding the suggested peer connections (again) to try and re-sync, but it froze up Chia (endless spinning circle for an hour) losing 10 plots that were over 50%, and causing me to nearly start chucking 12tb drives out the friggin' window. I, for the life of me, do not understand why farming and plotting have to be tied at the hip. I even went as far as to load Ubuntu this morning, but I couldn't even get Chia to even want to try to sync. It was worse. Really no point in wasting money\time on plots if you can't farm the damn things. It'd be different if this was an isolated issue, but it appears that a significant portion of users are experiencing the same problems. Please fix this on your next update.

@Jufik
Copy link

Jufik commented May 12, 2021

I agree, that's a major issue.
Is there any way I can contribute to this ?

@Trypt88
Copy link

Trypt88 commented May 12, 2021

I'm pretty frustrated with this issue also. There should be a file where we can put all those working nodes and have the Chia client runs thru the list on its own, instead of manually copy n paste all the IPs, and some of them don't ever work!!!!!

@KrakkenSmacken
Copy link

What might be a work around if someone is ambitious, would be to build a site like the one built for Pokemon Go that lets people post friends codes to. It might be better than https://chia.keva.app/ .

https://pokemongo.gishan.net/friends/codes/

Really though, this is a pretty core problem and needs to be addressed directly by the developers. I should be able to set a default connection number, and have the application manage the connections to keep that many connections up.

I personally wonder exactly what the Total Network Space would be if node didn't drop off so regularly.

@CY4N1DE
Copy link

CY4N1DE commented May 13, 2021

Same issue with me, after about 2 days of waiting I added the node-eu.chia.net:8444 to peers again and after about 30minutes the Sync started moving again. Its now finished

@gregoriob
Copy link

Tried many things in settings and agree with loads of comments. In the end, I found a cheap second hand pc with a better processor (from i3 to i7) actually less RAM but similar in every other way. The better processor managed to sync albeit slowly.
Other potential was that the i3 laptop was confusing itself with WiFi & Ethernet trying to run in parcelled.
If you read guides on farming with multiple PCs, it states that splitting farming & plotting separately is a better use of resource. Having one farmer with the blockchain synced up managing the farm & secondary (or more) plotting might be the work around.

@theunreal89
Copy link

theunreal89 commented May 19, 2021 via email

@gregoriob
Copy link

I have a Raspberry Pi 4 as a separate farmer (no plotting made on it, I
have a separate harvester) and the issue is still here on the farmer.

Stefano Scarpone

Il mer 19 mag 2021, 13:33 gregoriob @.***> ha scritto:

Tried many things in settings and agree with loads of comments. In the
end, I found a cheap second hand pc with a better processor (from i3 to i7)
actually less RAM but similar in every other way. The better processor
managed to sync albeit slowly.
Other potential was that the i3 laptop was confusing itself with WiFi &
Ethernet trying to run in parcelled.
If you read guides on farming with multiple PCs, it states that splitting
farming & plotting separately is a better use of resource. Having one
farmer with the blockchain synced up managing the farm & secondary (or
more) plotting might be the work around.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3298 (comment),
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA2UKYPVWWTG3UTRHIFNYX3TOOOW3ANCNFSM436C5RVQ
.

You don't need your farmer to sync from what I understand. You should only need one db in your internal network. Duplicating causes issues, just disable upnp on the farming node?

@theunreal89
Copy link

theunreal89 commented May 19, 2021 via email

@caspar-g
Copy link

My Raspberry Pi is both the farmer and full node, while I have a separate PC as an harvester (which is then not syncing with the blockchain but only provide the plots), I disabled UPNP and I have a manual port forwarding towards port 8444 of the Raspberry, indeed I can see nodes connecting to me correctly and the blockchain height is increasing but I never reach the top of the chain anyway. This was not happening since few days ago, so it's very weird. Il giorno mer 19 mag 2021 alle ore 14:11 gregoriob @.> ha scritto:

I have a Raspberry Pi 4 as a separate farmer (no plotting made on it, I have a separate harvester) and the issue is still here on the farmer. Stefano Scarpone Il mer 19 mag 2021, 13:33 gregoriob @.
> ha scritto: Tried many things in settings and agree with loads of comments. In the end, I found a cheap second hand pc with a better processor (from i3 to i7) actually less RAM but similar in every other way. The better processor managed to sync albeit slowly. Other potential was that the i3 laptop was confusing itself with WiFi & Ethernet trying to run in parcelled. If you read guides on farming with multiple PCs, it states that splitting farming & plotting separately is a better use of resource. Having one farmer with the blockchain synced up managing the farm & secondary (or more) plotting might be the work around. — You are receiving this because you commented. Reply to this email directly, view it on GitHub #3298 (comment) <#3298 (comment)> , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2UKYPVWWTG3UTRHIFNYX3TOOOW3ANCNFSM436C5RVQ . You don't need your farmer to sync from what I understand. You should only need one db in your internal network. Duplicating causes issues, just disable upnp on the farming node? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3298 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2UKYMWBDR7LZJMWWHQ5KTTOOTG7ANCNFSM436C5RVQ .
-- Stefano Scarpone

Same here. Exactly the same problems.

@davemerwin
Copy link

I actually brought this up a month ago on Keybase. Exact same problem. I thought it was the other nodes.

@krogla
Copy link

krogla commented May 20, 2021

+1

@kobbe
Copy link

kobbe commented May 20, 2021

I deleted the peer_table_node.sqlite and turned off disk sleep and now it works for me.

@nandesu
Copy link

nandesu commented May 21, 2021

Deleting .chia/mainnet/db/peer_table_node.sqlite worked for me as well. This forces the client to request new peers.

@gregoriob
Copy link

1.1.6 claims to resolve this now... let's see

@gregoriob
Copy link

It's off to a strong start!

@markusmazurczak
Copy link

Ja, right after the full-node start I got 7 connections to other full nodes. Catching up to the newest block is still a little bit slow in my opinion ...

My previous idea, that the disconnect has something to do with a internal full queue of bad peers was wrong.
My full nodes disconnects every night between 3:12 am and 3:27 am (which sounds a bit weird) but at the point when it starts trying to connect to only one peer for hours the internal queue was empty and even there were no blocked bad peers.

I am hoping that this release will fix the situation fingers crossed

@theunreal89
Copy link

theunreal89 commented May 21, 2021 via email

@asyslinux
Copy link

asyslinux commented May 21, 2021

Upgrading to 1.1.6 (and deleting .chia/mainnet/db/peer_table_node.sqlite too to be sure) finally solved the issue for me!

Confirm, i think solved, but i look next 24 hours for sure...

@fiveangle
Copy link
Contributor

Upgrading to 1.1.6 (and deleting .chia/mainnet/db/peer_table_node.sqlite too to be sure) finally solved the issue for me!

Confirm, i think solved, but i look next 24 hours for sure...

Not likely... I've observed stablized sync since last Fri 5/14, with no changes client-side across 12 systems at 4 different geographic locations. More likely, the exponential ramp-up of new zero-height clients signing-on/synching and dragging down all the other clients on the net has subsided. It's too bad no one took this issue seriously when it was occurring, else we might have been able to figure out a solution for when the net is faced with a similar issue. Maybe it will face that issue again... but we may never know. Let's hope not. With such a large number of clients at full-height now, it takes a much larger-and-larger numbers of zero-height clients connecting/min to hit that critical point, so hopefully those days are behind us. Still, it's worrisome in case a bad actor might be able to exploit the vulnerability, but given the architecture, I think we are safe. One would likely have resources to execute a 51% attack that would be much more useful than simply causing some <<50% of nodes to go out of sync.

@markusmazurczak
Copy link

Seems like the problem with that disconnect is solved. I have not have any issues for about 3 days now. No disconnect

@asyslinux
Copy link

asyslinux commented May 23, 2021

Seems like the problem with that disconnect is solved. I have not have any issues for about 3 days now. No disconnect

I am too, its now working like normally.

@simondarrens
Copy link

simondarrens commented May 23, 2021 via email

@PAVoutsinas
Copy link

I deleted the peer_table_node.sqlite and turned off disk sleep and now it works for me.

Deleting this file helped me as well. Thank you the post.

@nslsa661
Copy link

nslsa661 commented Jun 1, 2021 via email

@boner-cmd
Copy link

deleted this file, wallet still doesn't sync reliably at all

@boner-cmd
Copy link

in the time it took my node to sync from 0 to full, my wallet is at a block height of 25888 with no sign of budging.

@elforesto
Copy link

Sometimes your peer list will fill with nodes that are at a lower block height. Because of this, there's nobody that you can sync from, only sync to. The only solution is to restart your full node (which is taking an ever-increasing amount of time due to chain growth) and hope you get better nodes next time.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2021

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

@github-actions github-actions bot added the stale-issue flagged as stale and will be closed in 7 days if not updated label Jul 9, 2021
@github-actions
Copy link
Contributor

This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale-issue flagged as stale and will be closed in 7 days if not updated
Projects
None yet
Development

No branches or pull requests