Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TEP-0090: Fan Out TaskRuns #4990

Merged
merged 1 commit into from
Jun 17, 2022
Merged

TEP-0090: Fan Out TaskRuns #4990

merged 1 commit into from
Jun 17, 2022

Conversation

jerop
Copy link
Member

@jerop jerop commented Jun 16, 2022

Changes

TEP-0090: Matrix proposed executing a PipelineTask in parallel TaskRuns and Runs with substitutions from combinations of Parameters in a Matrix.

This change implements the fan out of TaskRuns from a PipelineTask with a Matrix. The fanned-out TaskRuns are executed in parallel.

/kind feature

Submitter Checklist

As the author of this PR, please check off the items in this checklist:

  • Docs included if any changes are user facing
  • Tests included if any functionality added or changed
  • Follows the commit message standard
  • Meets the Tekton contributor standards (including
    functionality, content, code)
  • Release notes block below has been filled in (if there are no user facing changes, use release note "NONE")

Release Notes

A `PipelineTask` with a `Matrix` is fanned out into parallel `TaskRuns` which are executed in parallel.

@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Jun 16, 2022
@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 16, 2022
@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

cc @abayer @dibyom @lbernick @pritidesai please take a look

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.2% -0.2

@jerop jerop force-pushed the tep-0090-createtaskruns branch from 4985acd to 66534ed Compare June 16, 2022 13:05
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop jerop force-pushed the tep-0090-createtaskruns branch 2 times, most recently from d7d8318 to f466791 Compare June 16, 2022 13:26
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/test pull-tekton-pipeline-go-coverage

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop jerop force-pushed the tep-0090-createtaskruns branch from f466791 to 1c85caa Compare June 16, 2022 13:49
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop jerop force-pushed the tep-0090-createtaskruns branch from 1c85caa to dd11cdb Compare June 16, 2022 14:06
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 85.8% -0.3
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop jerop force-pushed the tep-0090-createtaskruns branch 2 times, most recently from 40e99a5 to 9640ebf Compare June 16, 2022 14:36
@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/retest

@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

TLS handshake timeout 🤔

/retest

1 similar comment
@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

TLS handshake timeout 🤔

/retest

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 86.0% -0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/retest

@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.1% 86.0% -0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.5% 94.6% 0.1

@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-integration-tests

1 similar comment
@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/test pull-tekton-pipeline-alpha-integration-tests
/test pull-tekton-pipeline-integration-tests

@jerop jerop force-pushed the tep-0090-createtaskruns branch 3 times, most recently from 910ac46 to 8826b99 Compare June 16, 2022 21:33
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.4% 86.3% -0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.3% 94.4% 0.1

@jerop
Copy link
Member Author

jerop commented Jun 16, 2022

/test pull-tekton-pipeline-alpha-integration-tests

1 similar comment
@jerop
Copy link
Member Author

jerop commented Jun 17, 2022

/test pull-tekton-pipeline-alpha-integration-tests

@jerop jerop force-pushed the tep-0090-createtaskruns branch from 8826b99 to f0a9c38 Compare June 17, 2022 12:12
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.4% 86.3% -0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.3% 94.4% 0.1

@jerop
TEP-0090: Fan Out TaskRuns
cb1c733 
[TEP-0090: Matrix][tep-0090] proposed executing a `PipelineTask` in
parallel `TaskRuns` and `Runs` with substitutions from combinations
of `Parameters` in a `Matrix`.

This change implements the fan out of `TaskRuns` from a `PipelineTask`
with a `Matrix`. The fanned-out `TaskRuns` are executed in parallel.

[tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
@jerop jerop force-pushed the tep-0090-createtaskruns branch from f0a9c38 to cb1c733 Compare June 17, 2022 12:40
@tekton-robot
Copy link
Collaborator

The following is the coverage report on the affected files.
Say /test pull-tekton-pipeline-go-coverage to re-run this coverage report

File Old Coverage New Coverage Delta
pkg/reconciler/pipelinerun/pipelinerun.go 86.4% 86.3% -0.1
pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go 94.3% 94.4% 0.1

@jerop
Copy link
Member Author

jerop commented Jun 17, 2022

/test pull-tekton-pipeline-integration-tests

@tekton-robot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 17, 2022
@abayer
Copy link
Contributor

abayer commented Jun 17, 2022

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 17, 2022
@jerop
Copy link
Member Author

jerop commented Jun 17, 2022 •
edited
Loading

/test pull-tekton-pipeline-alpha-integration-tests

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.webhook.pipeline.tekton.dev": failed to call webhook: Post "https://tekton-pipelines-webhook.tekton-pipelines.svc:443/resource-validation?timeout=10s": context deadline exceeded in sidecar-ready-script

@jerop
Copy link
Member Author

jerop commented Jun 17, 2022

/test pull-pipeline-kind-k8s-v1-21-e2e

@abayer
Copy link
Contributor

abayer commented Jun 17, 2022

/test pull-tekton-pipeline-build-tests

1 similar comment
@jerop
Copy link
Member Author

jerop commented Jun 17, 2022

/test pull-tekton-pipeline-build-tests

@tekton-robot tekton-robot merged commit 5e53c61 into tektoncd:main Jun 17, 2022
@jerop jerop deleted the tep-0090-createtaskruns branch June 17, 2022 17:21
@pritidesai
Copy link
Member

thanks a bunch @jerop for this, appreciate all the efforts you have put in 🎉 ❤️ Can we please add an example and an e2e test for this feature, if they do not already exist?

@jerop
Copy link
Member Author

jerop commented Jun 21, 2022

thanks a bunch @jerop for this, appreciate all the efforts you have put in 🎉 ❤️ Can we please add an example and an e2e test for this feature, if they do not already exist?

@pritidesai fanning out of taskruns is not yet complete - at this point we create the taskruns but they are not yet passed to the pipelinerun as described in https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md#status

if i were to add examples now, the pipelinerun would stay running and timeout

opening the PRs implementing the status, which would include the examples and end-to-end tests

jerop added a commit to jerop/pipeline that referenced this pull request Jun 23, 2022
@jerop
Test reconciled PipelineRun from Pipeline with Matrix
ff9ba5a 
[TEP-0090: Matrix][tep-0090] proposed executing a `PipelineTask` in
parallel `TaskRuns` and `Runs` with substitutions from combinations
of `Parameters` in a `Matrix`.

In tektoncd#4990, we implemented
fanning out of `TaskRuns`, but they were not embedded in `PipelineRun`
status. Now that the minimal status are embedded through `ChildReferences`,
this change validates that the reconciled `PipelineRun` has the expected
`ChildReferences`.

[tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
@jerop jerop mentioned this pull request Jun 23, 2022
Merged
3 tasks
tekton-robot pushed a commit that referenced this pull request Jun 23, 2022
@jerop @tekton-robot
Test reconciled PipelineRun from Pipeline with Matrix
5d82fca 
[TEP-0090: Matrix][tep-0090] proposed executing a `PipelineTask` in
parallel `TaskRuns` and `Runs` with substitutions from combinations
of `Parameters` in a `Matrix`.

In #4990, we implemented
fanning out of `TaskRuns`, but they were not embedded in `PipelineRun`
status. Now that the minimal status are embedded through `ChildReferences`,
this change validates that the reconciled `PipelineRun` has the expected
`ChildReferences`.

[tep-0090]: https://github.com/tektoncd/community/blob/main/teps/0090-matrix.md
@jerop jerop mentioned this pull request Jun 28, 2022
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lbernick lbernick lbernick approved these changes

@afrittoli afrittoli Awaiting requested review from afrittoli

@pritidesai pritidesai Awaiting requested review from pritidesai

Assignees

@abayer abayer

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jerop @tekton-robot @abayer @pritidesai @lbernick