Parallelise artifact collection #2456
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DrJosh9000
force-pushed
the
faster-artifact-search
branch
3 times, most recently
from
3d7ab00 to
274b1b5
Compare
inez
approved these changes
Oct 27, 2023
|
Very interested in giving it a try! |
DrJosh9000
force-pushed
the
faster-artifact-search
branch
from
274b1b5 to
f858cd9
Compare
moskyb
approved these changes
Oct 29, 2023
agent/artifact_uploader.go
Outdated
|
|
||
| // Start N workers in a new context | ||
| wctx, cancel := context.WithCancelCause(ctx) | ||
| for i := 0; i < runtime.NumCPU(); i++ { |
There was a problem hiding this comment.
is runtime.NumCPU() being used here as "an arbitrary smallish number to parallelise by, which will be roughly proportional to the size of the box we're running on"? that's a totally valid use case, i just wanna be sure i'm reading right
There was a problem hiding this comment.
Yes. It's not a great heuristic though. We could create N goroutines and let the runtime figure it out. The code would be simpler, and unlike a server, the amount of work is fixed ahead of time, so it's not unreasonable. Actually, that idea is growing on me...
DrJosh9000
force-pushed
the
faster-artifact-search
branch
from
d191162 to
cf5edf7
Compare
DrJosh9000
force-pushed
the
faster-artifact-search
branch
from
cf5edf7 to
333e16d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It's possible to configure
Pathswith a lot of separate globs. It's not unreasonable that each of these could be of the form**/something, and each one requires walking the entire directory tree.go-zglobis not terribly slow because it uses a parallel walker under the hood (fastwalk). While I think about how to speed upzzglob, this is a reasonable first step.This also fixes a bug where
wdis replaced with"/"(orfilepath.VolumeName(absolutePath) + "/"on Windows) on the first glob path in the list that is is absolute, changingwdfor later glob paths that might not be absolute.