Fix Super Catchup Async Loop #10237
Merged
Fix Super Catchup Async Loop #10237
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nholland94
added
the
ci-build-me
mrmr1993
approved these changes
Feb 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In my thread timing work. I found that a particular async loop in the super catchup downloader was starving other jobs in the scheduler. The loop checks for a condition, waits for a signals related to events that may make that condition true, and then restarts. The issue was that the signals it was using to wait before checking again were based on pipes having messages. This is buggy because a pipe having a message is not the same as an event firing on the reader side of the message, which is what the code was assuming.
I added a
Bvar.tto the module to use as a proper event signaling primitive. This took the CPU usage of catchup down from 90%+ all the time (even when no catchup jobs were queued) to nearly 0% while no catchup jobs are running. Catchup still takes nearly 100% CPU during the initial catchup after bootstrap, but almost all of this time is spent building breadcrumbs instead of in the downloader module.