Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implementing pipeline level finally #2661

Merged
merged 1 commit into from
Jun 26, 2020
Merged

implementing pipeline level finally #2661

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
191 changes: 182 additions & 9 deletions docs/pipelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ weight: 3
- [Configuring execution results at the `Pipeline` level](#configuring-execution-results-at-the-pipeline-level)
- [Configuring the `Task` execution order](#configuring-the-task-execution-order)
- [Adding a description](#adding-a-description)
- [Adding `Finally` to the `Pipeline` (Preview)](#adding-finally-to-the-pipeline-preview)
- [Adding `Finally` to the `Pipeline`](#adding-finally-to-the-pipeline)
- [Code examples](#code-examples)

## Overview
Expand Down Expand Up @@ -530,10 +530,7 @@ In particular:

The `description` field is an optional field and can be used to provide description of the `Pipeline`.

## Adding `Finally` to the `Pipeline` (Preview)

_Finally type is available in the `Pipeline` but functionality is in progress. Final tasks are can be specified and
are validated but not executed yet._
## Adding `Finally` to the `Pipeline`

You can specify a list of one or more final tasks under `finally` section. Final tasks are guaranteed to be executed
in parallel after all `PipelineTasks` under `tasks` have completed regardless of success or error. Final tasks are very
Expand All @@ -553,10 +550,186 @@ spec:
Name: cleanup
```

_[PR #2661](https://github.com/tektoncd/pipeline/pull/2661) is implementing this new functionality by adding support to enable
final tasks along with workspaces and parameters. `PipelineRun` status is being updated to include execution status of
final tasks i.e. `PipelineRun` status is set to success or failure depending on execution of `PipelineTasks`, this status
remains same when all final tasks finishes successfully but is set to failure if any of the final tasks fail._
### Specifying `Workspaces` in Final Tasks

Finally tasks can specify [workspaces](workspaces.md) which `PipelineTasks` might have utilized
e.g. a mount point for credentials held in Secrets. To support that requirement, you can specify one or more
pritidesai marked this conversation as resolved.
Show resolved Hide resolved
`Workspaces` in the `workspaces` field for the final tasks similar to `tasks`.

```yaml
spec:
resources:
- name: app-git
type: git
workspaces:
- name: shared-workspace
tasks:
- name: clone-app-source
taskRef:
name: clone-app-repo-to-workspace
workspaces:
- name: shared-workspace
workspace: shared-workspace
resources:
inputs:
- name: app-git
resource: app-git
finally:
- name: cleanup-workspace
taskRef:
name: cleanup-workspace
workspaces:
- name: shared-workspace
workspace: shared-workspace
```

### Specifying `Parameters` in Final Tasks

Similar to `tasks`, you can specify [`Parameters`](tasks.md#specifying-parameters) in final tasks:

```yaml
spec:
tasks:
- name: tests
taskRef:
Name: integration-test
finally:
- name: report-results
taskRef:
Name: report-results
params:
- name: url
value: "someURL"
```

### `PipelineRun` Status with `finally`

With `finally`, `PipelineRun` status is calculated based on `PipelineTasks` under `tasks` section and final tasks.

Without `finally`:

| `PipelineTasks` under `tasks` | `PipelineRun` status | Reason |
| ----------------------------- | -------------------- | ------ |
| all `PipelineTasks` successful | `true` | `Succeeded` |
| one or more `PipelineTasks` [skipped](conditions.md) and rest successful | `true` | `Completed` |
| single failure of `PipelineTask` | `false` | `failed` |

With `finally`:

| `PipelineTasks` under `tasks` | Final Tasks | `PipelineRun` status | Reason |
| ----------------------------- | ----------- | -------------------- | ------ |
| all `PipelineTask` successful | all final tasks successful | `true` | `Succeeded` |
| all `PipelineTask` successful | one or more failure of final tasks | `false` | `Failed` |
| one or more `PipelineTask` [skipped](conditions.md) and rest successful | all final tasks successful | `true` | `Completed` |
| one or more `PipelineTask` [skipped](conditions.md) and rest successful | one or more failure of final tasks | `false` | `Failed` |
| single failure of `PipelineTask` | all final tasks successful | `false` | `failed` |
| single failure of `PipelineTask` | one or more failure of final tasks | `false` | `failed` |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ive just realized we have a similar table at https://github.com/tektoncd/pipeline/blob/master/docs/pipelineruns.md#monitoring-execution-status - i think it would make sense to include this information there (maybe link to it from here?) so ppl can go to one place (a section on status) to understand the status of a pipelinerun

Copy link
Member Author

@pritidesai pritidesai Jun 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, @afrittoli added it with PR #2677 😜 which was merged last week, I added a link to it. that table is more gear towards the kind of events being generated whereas this table is listing high level status. This table is highlighting difference between pipeline status for a pipeline with only DAG and a pipeline with DAG and finally


Overall, `PipelineRun` state transitioning is explained below for respective scenarios:

* All `PipelineTask` and final tasks are successful: `Started` -> `Running` -> `Succeeded`
* At least one `PipelineTask` skipped and rest successful: `Started` -> `Running` -> `Completed`
* One `PipelineTask` failed / one or more final tasks failed: `Started` -> `Running` -> `Failed`

Please refer to the [table](pipelineruns.md#monitoring-execution-status) under Monitoring Execution Status to learn about
what kind of events are triggered based on the `Pipelinerun` status.

### Known Limitations

### Specifying `Resources` in Final Tasks

Similar to `tasks`, you can use [PipelineResources](#specifying-resources) as inputs and outputs for
final tasks in the Pipeline. The only difference here is, final tasks with an input resource can not have a `from` clause
like a `PipelineTask` from `tasks` section. For example:

```yaml
spec:
tasks:
- name: tests
taskRef:
Name: integration-test
resources:
inputs:
- name: source
resource: tektoncd-pipeline-repo
outputs:
- name: workspace
resource: my-repo
finally:
- name: clear-workspace
taskRef:
Name: clear-workspace
resources:
inputs:
- name: workspace
resource: my-repo
from: #invalid
- tests
```

### Cannot configure the Final Task execution order

It's not possible to configure or modify the execution order of the final tasks. Unlike `Tasks` in a `Pipeline`,
all final tasks run simultaneously and start executing once all `PipelineTasks` under `tasks` have settled which means
no `runAfter` can be specified in final tasks.

### Cannot specify execution `Conditions` in Final Tasks

`Tasks` in a `Pipeline` can be configured to run only if some conditions are satisfied using `conditions`. But the
final tasks are guaranteed to be executed after all `PipelineTasks` therefore no `conditions` can be specified in
final tasks.

#### Cannot configure `Task` execution results with `finally`

Final tasks can not be configured to consume `Results` of `PipelineTask` from `tasks` section i.e. the following
example is not supported right now but we are working on adding support for the same (tracked in issue
[#2557](https://github.com/tektoncd/pipeline/issues/2557)).
pritidesai marked this conversation as resolved.
Show resolved Hide resolved

```yaml
spec:
tasks:
- name: count-comments-before
taskRef:
Name: count-comments
- name: add-comment
taskRef:
Name: add-comment
- name: count-comments-after
taskRef:
Name: count-comments
finally:
- name: check-count
taskRef:
Name: check-count
params:
- name: before-count
value: $(tasks.count-comments-before.results.count) #invalid
- name: after-count
value: $(tasks.count-comments-after.results.count) #invalid
```

#### Cannot configure `Pipeline` result with `finally`

Final tasks can emit `Results` but results emitted from the final tasks can not be configured in the
[Pipeline Results](#configuring-execution-results-at-the-pipeline-level). We are working on adding support for this
(tracked in issue [#2710](https://github.com/tektoncd/pipeline/issues/2710)).

```yaml
results:
- name: comment-count-validate
value: $(finally.check-count.results.comment-count-validate)
```

In this example, `PipelineResults` is set to:

```
"pipelineResults": [
{
"name": "comment-count-validate",
"value": "$(finally.check-count.results.comment-count-validate)"
}
],
```

## Code examples

Expand Down
187 changes: 187 additions & 0 deletions examples/v1beta1/pipelineruns/pipelinerun-with-final-tasks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
# Copied from https://github.com/tektoncd/catalog/blob/v1beta1/git/git-clone.yaml :(
pritidesai marked this conversation as resolved.
Show resolved Hide resolved
# This can be deleted after we add support to refer to the remote Task in a registry (Issue #1839) or
# add support for referencing task in git directly (issue #2298)
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone-from-catalog
spec:
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this workspace
params:
- name: url
description: git url to clone
type: string
- name: revision
description: git revision to checkout (branch, tag, sha, ref…)
type: string
default: master
- name: refspec
description: (optional) git refspec to fetch before checking out revision
default: ""
- name: submodules
description: defines if the resource should initialize and fetch the submodules
type: string
default: "true"
- name: depth
description: performs a shallow clone where only the most recent commit(s) will be fetched
type: string
default: "1"
- name: sslVerify
description: defines if http.sslVerify should be set to true or false in the global git config
type: string
default: "true"
- name: subdirectory
description: subdirectory inside the "output" workspace to clone the git repo into
type: string
default: ""
- name: deleteExisting
description: clean out the contents of the repo's destination directory (if it already exists) before trying to clone the repo there
type: string
default: "false"
- name: httpProxy
description: git HTTP proxy server for non-SSL requests
type: string
default: ""
- name: httpsProxy
description: git HTTPS proxy server for SSL requests
type: string
default: ""
- name: noProxy
description: git no proxy - opt out of proxying HTTP/HTTPS requests
type: string
default: ""
results:
- name: commit
description: The precise commit SHA that was fetched by this Task
steps:
- name: clone
image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.12.1
script: |
CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"

cleandir() {
# Delete any existing contents of the repo directory if it exists.
#
# We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
# or the root of a mounted volume.
if [[ -d "$CHECKOUT_DIR" ]] ; then
# Delete non-hidden files and directories
rm -rf "$CHECKOUT_DIR"/*
# Delete files and directories starting with . but excluding ..
rm -rf "$CHECKOUT_DIR"/.[!.]*
# Delete files and directories starting with .. plus any other character
rm -rf "$CHECKOUT_DIR"/..?*
fi
}

if [[ "$(params.deleteExisting)" == "true" ]] ; then
cleandir
fi

test -z "$(params.httpProxy)" || export HTTP_PROXY=$(params.httpProxy)
test -z "$(params.httpsProxy)" || export HTTPS_PROXY=$(params.httpsProxy)
test -z "$(params.noProxy)" || export NO_PROXY=$(params.noProxy)

/ko-app/git-init \
-url "$(params.url)" \
-revision "$(params.revision)" \
-refspec "$(params.refspec)" \
-path "$CHECKOUT_DIR" \
-sslVerify="$(params.sslVerify)" \
-submodules="$(params.submodules)" \
-depth "$(params.depth)"
cd "$CHECKOUT_DIR"
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)

---

# Task to cleanup shared workspace
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: cleanup-workspace
spec:
workspaces:
# Shared workspace where git repo is cloned
- name: source
steps:
- name: check-application-dir-has-source
pritidesai marked this conversation as resolved.
Show resolved Hide resolved
image: ubuntu
script: |
if [ ! -d "$(workspaces.source.path)/application/" ]; then
echo "Something went wrong and could not find application source under $(workspaces.source.path)/application/"
exit 1
fi
- name: cleanup-workspace
image: ubuntu
script: |
rm -rf $(workspaces.source.path)/application/
- name: verify-application-dir-has-gone
image: ubuntu
script: |
if [ -d "$(workspaces.source.path)/application/" ]; then
echo "Something went wrong cleaning up and the application source still exists under $(workspaces.source.path)/application/"
exit 1
fi
---

# Pipeline to clone repo into shared workspace and cleanup the workspace after done
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: clone-cleanup-workspace
spec:
workspaces:
# common workspace where git repo is cloned and needs to be cleanup after done
- name: git-source
tasks:
# Clone app repo to workspace
- name: clone-app-repo
taskRef:
name: git-clone-from-catalog
params:
- name: url
value: https://github.com/tektoncd/community.git
- name: subdirectory
value: application
workspaces:
- name: output
workspace: git-source
finally:
# Cleanup workspace
- name: cleanup
taskRef:
name: cleanup-workspace
workspaces:
- name: source
workspace: git-source
---

# PipelineRun to execute pipeline - clone-into-workspace-and-cleanup-workspace
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: write-and-cleanup-workspace
spec:
pipelineRef:
name: clone-cleanup-workspace
serviceAccountName: 'default'
workspaces:
- name: git-source
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
Loading