Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bazel test Throttle resources for CI #196

Closed
wants to merge 1 commit into from
Closed

Add bazel test Throttle resources for CI #196

wants to merge 1 commit into from

Conversation

yxun
Copy link
Member

@yxun yxun commented Aug 30, 2022

Signed-off-by: Yuanlin [email protected]

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

@yxun
Copy link
Member Author

yxun commented Aug 30, 2022

When I run maistra-2.2 maistra/envoy maistra/run-ci.sh in an OCP pod, I set the env variable CI="true" in the container.
At the beginning of the log , I can see --config=ci-config

++ '[' -n true ']'
++ COMMON_FLAGS+=' --config=ci-config '
++ '[' -n '' ']'
++ '[' -n '' ']'

However, in the middle of the run-ci.sh exection, my pod CPU jumped and reached request limits.Then OCP killed an integration test execution when it requested more CPU than the limit.

image

From the reference, it looks like we only set the resource throttle in the bazel build ... part. But we haven't set any resource throttle in the bazel test ... part yet.
https://bazel.build/reference/command-line-reference

@yxun yxun requested a review from oschaaf August 30, 2022 22:03
@oschaaf
Copy link
Contributor

oschaaf commented Aug 31, 2022

We discussed this on Slack earlier: I feel that tuning these resource limits should be done only when our CI actually needs it. Having said that, maybe a bazel comment indicating that said config line only throttles build and not test execution is worth it?

I'm curious if the CI failure is a flake of some sorts, I have a hard time associating them to this change.. so:

/retest

@maistra-bot
Copy link

@yxun: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
envoy-unit-2.2 e465a39 link true /test unit

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@@ -20,6 +20,9 @@ build:aarch64 --linkopt=-fuse-ld=lld
build:ci-config --local_ram_resources=12288
build:ci-config --local_cpu_resources=6
build:ci-config --jobs=3
test:ci-config --local_ram_resources=12288
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be necessary as test inherits is configuration from build: https://bazel.build/run/bazelrc#option-defaults

@oschaaf
Copy link
Contributor

oschaaf commented Sep 1, 2022

Closing this: with earlier testing, me and @jwendell have observed that the change proposed here actually is a no-op (as suggested in a comment above).

@oschaaf oschaaf closed this Sep 1, 2022
@yxun
Copy link
Member Author

yxun commented Sep 6, 2022

https://bazel.build/docs/user-manual#running-tests
"By default, this command performs simultaneous build and test activity"
That might explain why the cpu usage is over the config value 6.
I will try some workaround thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants