Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

stuck at SenderRecovery #7858

Closed
1 task done
mrabino1 opened this issue Apr 25, 2024 · 6 comments
Closed
1 task done

stuck at SenderRecovery #7858

mrabino1 opened this issue Apr 25, 2024 · 6 comments
Labels
A-staged-sync Related to staged sync (pipelines and stages) C-bug An unexpected or incorrect behavior

Comments

@mrabino1
Copy link

Describe the bug

see logs below. seemingly stuck at SenderRecovery

Steps to reproduce

install beta.6 .. sync full from start.. wait

Node logs

$ tail -f reth.log 
2024-04-25T05:30:37.727450Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.732667Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:30:37.732880Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.869789Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:37.870105Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:38.601058Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:30:40.224789Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:44.239314Z DEBUG discv4: Failed to send datagram. to=128.10.126.23:0 err=Invalid argument (os error 22)
2024-04-25T05:30:44.602015Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:45.421767Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.227048Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.230649Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:30:49.230851Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.361554Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:49.361853Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:30:52.226295Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.056854Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.059692Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:01.059895Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.251420Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:01.251760Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:03.601077Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:31:04.228211Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.415023Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.415960Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.785010Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.789528Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:12.789777Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.923689Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:12.924015Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:16.228813Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.111408Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.114043Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:25.114300Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.244952Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:25.245304Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:28.229315Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:28.601081Z  INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=unknown
2024-04-25T05:31:37.210277Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.213663Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-25T05:31:37.213882Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.346096Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:37.346430Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-25T05:31:40.231135Z DEBUG jsonrpsee-server: Accepting new connection 1/500

Platform(s)

Linux (x86)

What version/commit are you on?

beta.6

What database version are you on?

beta.6

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

  • I agree to follow the Code of Conduct
@mrabino1 mrabino1 added C-bug An unexpected or incorrect behavior S-needs-triage This issue needs to be labelled labels Apr 25, 2024
@mrabino1
Copy link
Author

a joyful
sudo systemctl restart reth.service

has seemingly kicked it.. and allowed it to continue..

@mrabino1
Copy link
Author

here are the logs right before things stopped progressing... fwiw

2024-04-24T20:01:08.442017Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333025538..1333125538
2024-04-24T20:01:08.442030Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333125538..1333225538
2024-04-24T20:01:08.442184Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333125538..1333225538
2024-04-24T20:01:09.583157Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333125538..1333225538
2024-04-24T20:01:09.583170Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333225538..1333325538
2024-04-24T20:01:09.583305Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333225538..1333325538
2024-04-24T20:01:10.680326Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333225538..1333325538
2024-04-24T20:01:10.680339Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333325538..1333425538
2024-04-24T20:01:10.680561Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333325538..1333425538
2024-04-24T20:01:11.747709Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333325538..1333425538
2024-04-24T20:01:11.747720Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333425538..1333525538
2024-04-24T20:01:11.747856Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333425538..1333525538
2024-04-24T20:01:12.421821Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:12.422400Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:12.709291Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:12.781587Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:12.911447Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-24T20:01:12.911740Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:12.914792Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333425538..1333525538
2024-04-24T20:01:12.914800Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333525538..1333625538
2024-04-24T20:01:12.914954Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333525538..1333625538
2024-04-24T20:01:13.189575Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:13.193612Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:14.284530Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333525538..1333625538
2024-04-24T20:01:14.284543Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333625538..1333725538
2024-04-24T20:01:14.284737Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333625538..1333725538
2024-04-24T20:01:14.841526Z DEBUG net::session: pending session timed out remote_addr=213.225.253.90:30303 direction=Outgoing(0x40456c1d52acecaa4f964110be1a207d99f1af8b98022d6e5b48f41c1010e7f0226ea2f406d2d0008dc68abe119436a298b35a08d17516ff578170db2159c168)
2024-04-24T20:01:15.682204Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333625538..1333725538
2024-04-24T20:01:15.682216Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333725538..1333825538
2024-04-24T20:01:15.682358Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333725538..1333825538
2024-04-24T20:01:16.913738Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333725538..1333825538
2024-04-24T20:01:16.913751Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333825538..1333925538
2024-04-24T20:01:16.913917Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333825538..1333925538
2024-04-24T20:01:18.202049Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333825538..1333925538
2024-04-24T20:01:18.202062Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1333925538..1334025538
2024-04-24T20:01:18.202241Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1333925538..1334025538
2024-04-24T20:01:19.601992Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1333925538..1334025538
2024-04-24T20:01:19.602005Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334025538..1334125538
2024-04-24T20:01:19.602199Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334025538..1334125538
2024-04-24T20:01:20.577321Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x15c9b7dd86939afdcff892f85e70b99d3733877ecc44ed1e1b865f65a02bb44693ec2cae2148f51803823542e7728a9996c365cfbc37bb72ef55bee7e4fc9a95
2024-04-24T20:01:20.894949Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334025538..1334125538
2024-04-24T20:01:20.894962Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334125538..1334225538
2024-04-24T20:01:20.895141Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334125538..1334225538
2024-04-24T20:01:22.163950Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334125538..1334225538
2024-04-24T20:01:22.163963Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334225538..1334325538
2024-04-24T20:01:22.164252Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334225538..1334325538
2024-04-24T20:01:23.363315Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334225538..1334325538
2024-04-24T20:01:23.363329Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334325538..1334425538
2024-04-24T20:01:23.363500Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334325538..1334425538
2024-04-24T20:01:24.577785Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334325538..1334425538
2024-04-24T20:01:24.577797Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334425538..1334525538
2024-04-24T20:01:24.578004Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334425538..1334525538
2024-04-24T20:01:24.724788Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:25.312298Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:25.904397Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-24T20:01:25.905547Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:25.906369Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334425538..1334525538
2024-04-24T20:01:25.906377Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334525538..1334625538
2024-04-24T20:01:25.906611Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334525538..1334625538
2024-04-24T20:01:26.289952Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:26.293731Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:27.286999Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334525538..1334625538
2024-04-24T20:01:27.287013Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334625538..1334725538
2024-04-24T20:01:27.287340Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334625538..1334725538
2024-04-24T20:01:28.602029Z INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=3h 24m 24s
2024-04-24T20:01:28.614645Z DEBUG sync::stages::sender_recovery: Finished recovering senders batch tx_range=1334625538..1334725538
2024-04-24T20:01:28.614654Z DEBUG sync::stages::sender_recovery: Recovering senders batch tx_range=1334725538..1334825538
2024-04-24T20:01:28.614947Z DEBUG sync::stages::sender_recovery: Appending recovered senders to the database tx_range=1334725538..1334825538
2024-04-24T20:01:36.503277Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:36.508943Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-24T20:01:36.509133Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:36.637294Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:36.637611Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:36.712185Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:48.711811Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:48.713391Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:48.715257Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-24T20:01:48.715423Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:48.839259Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:48.839587Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:01:53.601559Z INFO reth::cli: Status connected_peers=100 freelist=8 stage=SenderRecovery checkpoint=13476639 target=19724655 stage_progress=56.70% stage_eta=3h 23m 59s
2024-04-24T20:02:00.715043Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:02:01.251175Z DEBUG jsonrpsee-server: Accepting new connection 1/500
2024-04-24T20:02:01.254890Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"

@emhane
Copy link
Member

emhane commented May 8, 2024

you can try to postpone pruning #7500 (comment). I would also turn of the rpc server for initial sync. @mrabino1

@emhane emhane added A-staged-sync Related to staged sync (pipelines and stages) and removed S-needs-triage This issue needs to be labelled labels May 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

This issue is stale because it has been open for 21 days with no activity.

@github-actions github-actions bot added the S-stale This issue/PR is stale and will close with no further activity label May 30, 2024
@emhane
Copy link
Member

emhane commented May 31, 2024

did syncing work out for you in the end @mrabino1 ?

@github-actions github-actions bot removed the S-stale This issue/PR is stale and will close with no further activity label Jun 1, 2024
@mrabino1
Copy link
Author

mrabino1 commented Jun 1, 2024

y. i will close. looking forward this being out of beta !

@mrabino1 mrabino1 closed this as completed Jun 1, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Reth Tracker Jun 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-staged-sync Related to staged sync (pipelines and stages) C-bug An unexpected or incorrect behavior
Projects
Reth Tracker
Archived in project
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mrabino1 @emhane