Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

workaround for eventstream disconnects #116

Merged
merged 1 commit into from
Aug 26, 2024

Conversation

pk910
Copy link
Member

@pk910 pk910 commented Aug 26, 2024

Workaround for the random repeatedly occuring mass eventstream disconnects:

time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=lodestar-reth-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=nimbus-besu-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=nimbus-geth-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=lodestar-nethermind-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=lodestar-besu-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=nimbus-nethermind-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=nimbus-reth-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=lodestar-geth-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=lodestar-ethereumjs-1 service=cl-pool
time="2024-08-26T11:12:00Z" level=warning msg="beacon block stream error: stream error: stream ID 25; INTERNAL_ERROR; received from peer" client=nimbus-ethereumjs-1 service=cl-pool

This affects lodestar & nimbus clients only (previously nimbus only). All these connections seem to be killed at the exact same time (at full minutes like 11:12:00 above), and even from different explorer instances.

It seems to be a go bug related to http2: golang/go#51323

This PR silently drops this specific error and reconnects the affected event stream immediately without the usual reconnect delay.

@pk910 pk910 merged commit f51115d into master Aug 26, 2024
1 check passed
@pk910 pk910 deleted the pk910/fix-eventstream-disconnects branch September 9, 2024 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants