replace ForkJoinPool with fixed thread pool #4552
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
oracle-contributor-agreement
bot
added
the
OCA Verified
vladak
changed the title
|
Repeated the measurements using the same setup but with statsd metrics collection enabled. Also, I disabled Tomcat to avoid suggester rebuild interfering with the indexing (in terms of CPU load). before (2 days 06:08:44): after (1 day 20:08:07): Now, there is something about the gaps of CPU usage in the second indexing phase, and this should certainly be investigated, however given it's overwhelmingly better that the previous solution makes this easy win. |
|
I ran the index from scratch once again, this time watching the average load with a Python program. When the average load for the last minute dropped below 10, it grabbed Basically, at the point the tasks being processed were all pertaining to a single project that consisted of a Subversion repository. The annotation cache was enabled and such processing is more I/O than CPU bound for Subversion, hence the decrease. While it would be nicer if the graph was smoother, e.g. by somehow mixing the jobs from various projects more evenly, I don't see it as a blocker for integration, esp. considering that annotation cache is disabled by default. |
opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java
Show resolved
Hide resolved
opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java
Outdated
Show resolved
Hide resolved
opengrok-indexer/src/main/java/org/opengrok/indexer/index/IndexDatabase.java
Outdated
Show resolved
Hide resolved
ginoaugustine
approved these changes
Mar 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change replaces the
ForkJoinPoolused for distributingaddFile()tasks in the 2nd phase of indexer processing with fixed thread pool. In my case:-J=-Xmx48g -J=-serverusing--renamedHistory 'on' -r dirbased -G -m '256' --threads 48 --historyThreads 48 --historyFileThreads 32and annotation cache enabled for all projectsit shaves some 8+ hours from the total indexing time:
before:
after:
The distribution of the tasks is not as uniform - more often than not a single project occupies the fixed thread pool (while always using it to its full capacity unlike the
ForkJoin), however the sub-optimal scheduling is gone. Also, I don't like the hazy promise of customForkJoinPoolto be used forparallelStream()processing.