Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop resource requests: from gohelloworld. #4549

Merged
merged 1 commit into from
Feb 7, 2022

Conversation

mattmoor
Copy link
Member

@mattmoor mattmoor commented Feb 5, 2022

Resource requests are used to decide whether a pod may be scheduled on a node. In some of our downstream testing we have been seeing a rash of flakes in TestHelmDeployPipelineRun which manifest as follows.

The pipeline itself times out waiting for the test to complete:

2022-02-05T03:06:49.873527563Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:51.905640413Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:54.013351746Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:55.814048343Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:55.822633914Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:56.162347025Z stderr F Error: timed out waiting for the condition
2022-02-05T03:06:56.165538751Z stderr F helm.go:81: [debug] timed out waiting for the condition

However, a brief while later:

2022-02-05T03:08:36.385531093Z stderr F 2022/02/05 03:08:36 Hello world sample started.
2022-02-05T03:08:37.246650188Z stderr F 2022/02/05 03:08:37 Hello world received a request.
2022-02-05T03:08:45.852975472Z stderr F 2022/02/05 03:08:45 Hello world received a request.
2022-02-05T03:08:45.856908405Z stderr F 2022/02/05 03:08:45 Hello world received a request.
2022-02-05T03:08:55.851238051Z stderr F 2022/02/05 03:08:55 Hello world received a request.
2022-02-05T03:08:55.855837089Z stderr F 2022/02/05 03:08:55 Hello world received a request.

I believe that the core issue here is that it times out because the pod cannot be scheduled, likely because the KinD cluster we're on is very busy.

An alternative here might be to (dramatically) lower the resource requests, but I'm not sure they are providing any real value today.

Related: #4455

Changes

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Docs included if any changes are user facing
  • Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes

Resource requests are used to decide whether a pod may be scheduled on a node.  In some of our downstream testing we have been seeing a rash of flakes in `TestHelmDeployPipelineRun` which manifest as follows.

The pipeline itself times out waiting for the test to complete:
```
2022-02-05T03:06:49.873527563Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:51.905640413Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:54.013351746Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:55.814048343Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:55.822633914Z stderr F wait.go:244: [debug] Deployment is not ready: arendelle-7wpjd/gohelloworld-chart. 0 out of 1 expected pods are ready
2022-02-05T03:06:56.162347025Z stderr F Error: timed out waiting for the condition
2022-02-05T03:06:56.165538751Z stderr F helm.go:81: [debug] timed out waiting for the condition
```

However, a brief while later:
```
2022-02-05T03:08:36.385531093Z stderr F 2022/02/05 03:08:36 Hello world sample started.
2022-02-05T03:08:37.246650188Z stderr F 2022/02/05 03:08:37 Hello world received a request.
2022-02-05T03:08:45.852975472Z stderr F 2022/02/05 03:08:45 Hello world received a request.
2022-02-05T03:08:45.856908405Z stderr F 2022/02/05 03:08:45 Hello world received a request.
2022-02-05T03:08:55.851238051Z stderr F 2022/02/05 03:08:55 Hello world received a request.
2022-02-05T03:08:55.855837089Z stderr F 2022/02/05 03:08:55 Hello world received a request.
```

I believe that the core issue here is that it times out because the pod cannot be scheduled, likely because the KinD cluster we're on is very busy.

An alternative here might be to (dramatically) lower the resource requests, but I'm not sure they are providing any real value today.

Related: tektoncd#4455
@tekton-robot tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 5, 2022
@tekton-robot tekton-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 5, 2022
@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 7, 2022
Copy link
Member

@vdemeester vdemeester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 7, 2022
@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: imjasonh, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [imjasonh,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot merged commit ed4200b into tektoncd:main Feb 7, 2022
@mattmoor mattmoor deleted the drop-resource-requests branch February 7, 2022 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants