Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run make in parallel instead of adding cpus to tests #2892

Closed
wants to merge 6 commits into from
Closed

Run make in parallel instead of adding cpus to tests #2892

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5a89801
Run make in parallel instead of adding cpus to tests
Madrigal Nov 8, 2024
20d4827
[experimental] test results with a go.work file
Madrigal Nov 11, 2024
f348236
Add double quotes on go list and remove the cpu flag cap
Madrigal Nov 11, 2024
808dd34
Explicitely set the number of CPU cores to use
Madrigal Nov 12, 2024
c91065e
Last test: just see what happens if we don't add module concurrency
Madrigal Nov 12, 2024
84332ff
Update concurrency since Windows tests were actually slower with conc…
Madrigal Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/go.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:
- 'feat-**'

env:
EACHMODULE_CONCURRENCY: 2
EACHMODULE_CONCURRENCY: 4
SMITHY_GO_REPOSITORY: ${{ github.event.pull_request.head.repo.owner.login }}/smithy-go
GIT_PAT: ${{ secrets.CI_GIT_PAT}}

Expand Down
24 changes: 23 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,25 @@ unit-race-modules-%:
&& go run . -p $(subst _,/,$(subst unit-race-modules-,,$@)) ${EACHMODULE_FLAGS} \
"go vet ${BUILD_TAGS} --all ./..." \
"go test ${BUILD_TAGS} ${RUN_NONE} ./..." \
"go test -timeout=2m ${UNIT_TEST_TAGS} -race -cpu=4 ./..."
"go test -timeout=2m ${UNIT_TEST_TAGS} -race ./..."

# EXPERIMENT will explore more if the results are good
# Basically instead of invoking "go test" n times per module (we have around 400 of them)
# call "go test" will all modules as arguments. This however is not as straightforward since
# this is a monorepo, so we have multiple modules that are not visible from root.
# Doing this needs a `go.work` file that there's a hot debate on whether it should be included into VCS or not
# So, before exploring this, ensure that the benefits are there
experimental-module:
go list -f "{{.Dir}}/..." -m | xargs go vet ${BUILD_TAGS} --all && \
go list -f "{{.Dir}}/..." -m | xargs go test ${BUILD_TAGS} ${RUN_NONE} && \
go list -f "{{.Dir}}/..." -m | xargs go test -timeout=2m ${UNIT_TEST_TAGS} -race -cpu=4

# same as regular but no race test
experimental-module-no-race:
go list -f "{{.Dir}}/..." -m | xargs go vet ${BUILD_TAGS} --all && \
go list -f "{{.Dir}}/..." -m | xargs go test ${BUILD_TAGS} ${RUN_NONE} && \
go list -f "{{.Dir}}/..." -m | xargs go test -timeout=2m ${UNIT_TEST_TAGS} -cpu=4


unit-modules-%:
@# unit command that uses the pattern to define the root path that the
Expand Down Expand Up @@ -510,6 +528,10 @@ vet-modules-%:
&& go run . -p $(subst _,/,$(subst vet-modules-,,$@)) ${EACHMODULE_FLAGS} \
"go vet ${BUILD_TAGS} --all ./..."

experimental-vet:
go list -f '{{.Dir}}/...' -m | xargs go vet ${BUILD_TAGS} --all


sdkv1check:
@echo "Checking for usage of AWS SDK for Go v1"
@sdkv1usage=`go list -test -f '''{{ if not .Standard }}{{ range $$_, $$name := .Imports }} * {{ $$.ImportPath }} -> {{ $$name }}{{ print "\n" }}{{ end }}{{ range $$_, $$name := .TestImports }} *: {{ $$.ImportPath }} -> {{ $$name }}{{ print "\n" }}{{ end }}{{ end}}''' ./... | sort -u | grep '''/aws-sdk-go/'''`; \
Expand Down
Loading
Loading