Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tune release-blocking kind job resources down #19081

Merged

Conversation

spiffxp
Copy link
Member

@spiffxp spiffxp commented Sep 1, 2020

The kind release-blocking prowjobs have been ending up in error
state over the past few week days, often due to pods being unable
to schedule due to insufficient CPU. We've been seeing increased
PR traffic due to v1.19 getting released, and a concerted attempt
to clear the backlog of v1.20 PRs that were waiting for code thaw.

The kind release-blocking jobs were the only jobs migrated to
k8s-infra-prow-build that reserved 7300m CPU. The next highest
cpu limit is 7, used by the verify presubmit and bazel-test periodic
jobs. These jobs have not been ending up in error state.

I was originally going to match the kind presubmit jobs, which use
4 cpu, but they have been failing a lot today during times I would
correlate with high PR volume. I'm still tuning the memory to match
based on graphs in the issue linked to this PR.

Hoping to address #19080

The kind release-blocking prowjobs have been ending up in error
state over the past few week days, often due to pods being unable
to schedule due to insufficient CPU. We've been seeing increased
PR traffic due to v1.19 getting released, and a concerted attempt
to clear the backlog of v1.20 PRs that were waiting for code thaw.

The kind release-blocking jobs were the only jobs migrated to
k8s-infra-prow-build that reserved 7300m CPU. The next highest
cpu limit is 7, used by the verify presubmit and bazel-test periodic
jobs. These jobs have not been ending up in error state.

I was originally going to match the kind presubmit jobs, which use
4 cpu, but they have been failing a lot today during times I would
correlate with high PR volume. I'm still tuning the memory to match
based on graphs in the issue linked to this PR.
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. area/config Issues or PRs related to code in /config area/jobs area/release-eng Issues or PRs related to the Release Engineering subproject sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 1, 2020
@spiffxp
Copy link
Member Author

spiffxp commented Sep 1, 2020

/cc @BenTheElder
FYI @kubernetes/ci-signal

Copy link
Contributor

@hasheddan hasheddan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 1, 2020
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hasheddan, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 278d2f1 into kubernetes:master Sep 1, 2020
@k8s-ci-robot k8s-ci-robot added this to the v1.20 milestone Sep 1, 2020
@k8s-ci-robot
Copy link
Contributor

@spiffxp: Updated the job-config configmap in namespace default at cluster default using the following files:

  • key kind-release-blocking.yaml using file config/jobs/kubernetes-sigs/kind/kind-release-blocking.yaml
  • key 1.16.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.16.yaml
  • key 1.17.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.17.yaml
  • key 1.18.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.18.yaml
  • key 1.19.yaml using file config/jobs/kubernetes/sig-release/release-branch-jobs/1.19.yaml

In response to this:

The kind release-blocking prowjobs have been ending up in error
state over the past few week days, often due to pods being unable
to schedule due to insufficient CPU. We've been seeing increased
PR traffic due to v1.19 getting released, and a concerted attempt
to clear the backlog of v1.20 PRs that were waiting for code thaw.

The kind release-blocking jobs were the only jobs migrated to
k8s-infra-prow-build that reserved 7300m CPU. The next highest
cpu limit is 7, used by the verify presubmit and bazel-test periodic
jobs. These jobs have not been ending up in error state.

I was originally going to match the kind presubmit jobs, which use
4 cpu, but they have been failing a lot today during times I would
correlate with high PR volume. I'm still tuning the memory to match
based on graphs in the issue linked to this PR.

Hoping to address #19080

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@spiffxp spiffxp deleted the tune-down-release-blocking-kind-jobs branch September 1, 2020 03:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/config Issues or PRs related to code in /config area/jobs area/release-eng Issues or PRs related to the Release Engineering subproject cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants