Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RESOLVED] ⚠️Production outage (partial): bot ceased receiving events for certain servers from Discord after about 2022-12-14 11:01 UTC #246

Closed
kevinlul opened this issue Dec 15, 2022 · 4 comments
Closed

[RESOLVED] ⚠️Production outage (partial): bot ceased receiving events for certain servers from Discord after about 2022-12-14 11:01 UTC #246

kevinlul opened this issue Dec 15, 2022 · 4 comments
Labels
production Incidents from the running Bastion#1870 instance

Comments

@kevinlul
Copy link
Contributor

Resolved 2022-12-14T12:27:11.230Z by container reboot.

Certain shards disconnected from the Discord gateway and either failed to reconnect, or Discord ceased sending events for those shards to the bot. This resulted in a partial outage for about ~1.5 hours until operator intervention. Some servers would be affected and others would not be, depending on the shard they were assigned. Shard 0 is the only one that was certainly not affected by this, which includes direct messages.

Log excerpt:

2022-12-14T11:00:06.776Z bot:info:abdeploy Updating from /var/local/bastion/abdeploy.json
2022-12-14T11:00:06.777Z bot:info:abdeploy Read 2 entries
USER SEARCHES
2022-12-14T11:01:00.760Z bot:info:events Shard 1 reconnecting
2022-12-14T11:01:00.945Z bot:info:events Shard 1 resumed: 1 events replayed
2022-12-14T11:01:05.761Z bot:info:events Shard 1 reconnecting
2022-12-14T11:01:25.709Z bot:info:events Shard 5 reconnecting
2022-12-14T11:01:25.971Z bot:info:events Shard 5 resumed: 38 events replayed
2022-12-14T11:01:30.711Z bot:info:events Shard 5 reconnecting
2022-12-14T11:02:18.611Z bot:info:events Shard 4 reconnecting
2022-12-14T11:02:22.038Z bot:info:events Shard 6 reconnecting
2022-12-14T11:02:24.778Z bot:notify:events Shard 4 ready
2022-12-14T11:02:27.827Z bot:notify:events Shard 6 ready
2022-12-14T11:02:43.612Z bot:info:events Shard 4 reconnecting
2022-12-14T11:02:47.039Z bot:info:events Shard 6 reconnecting
2022-12-14T11:03:05.482Z bot:info:events Shard 2 reconnecting
USER SEARCHES
2022-12-14T12:26:21.141Z OPERATOR ATTEMPTS CONTAINER REBOOT
@kevinlul kevinlul added the production Incidents from the running Bastion#1870 instance label Dec 15, 2022
@kevinlul
Copy link
Contributor Author

kevinlul commented Dec 20, 2022 •
edited
Loading

Possible root cause for all five incidents: discordjs/discord.js#8486

Past incidents: #199, #204, #212, #233

@kevinlul
Copy link
Contributor Author

What a patch https://github.com/NeonWizard/sockbot-discord/blob/main/src/modules/wspatch.ts

kevinlul added a commit that referenced this issue Feb 5, 2023
@kevinlul
Prevent setInterval timers from keeping the event loop running
3fd629c 
https://nodejs.org/docs/latest-v18.x/api/timers.html#timeoutunref

May help with #246
@kevinlul
Copy link
Contributor Author

Another incident around 2023/03/08 15:57 ET

@kevinlul
Copy link
Contributor Author

Fixed in a079568 (discordjs/discord.js#8989 is included in 14.8.0 and closed discordjs/discord.js#8486) unless proven otherwise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
production Incidents from the running Bastion#1870 instance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kevinlul