Skip to content

Commit

Permalink
Implement WhenExpressions
Browse files Browse the repository at this point in the history
Adding `WhenExpressions` used to efficiently specify guarded execution
of `Tasks`, without spinning up new pods. We use `WhenExpressions` to
avoid adding an opinionated and complex expression language to the Tekton
API to ensure Tekton can be supported by as many systems as possible.

The components of `WhenExpressions` are `Input`, `Operator`
and `Values`:
- `Input` is the input for the `Guard` checking which can be static
inputs or variables, such as `Parameters` or `Results`.
- `Values` is an array of string values. The `Values` array must be
non-empty. It can contain static values or variables (`Parameters` or
`Results`).
- `Operator` represents an `Input`'s relationship to a set of `Values`.
`Operators` we will use in `WhenExpressions` are `In` and `NotIn`.

The declared `WhenExpressions` are evaluated before the `Task` is run.
If all the `WhenExpressions` evaluate to `True`, the `Task` is run.
If any of the `WhenExpressions` evaluate to `False`, the `Task` is
skipped.

Further details in [Conditions Beta TEP](https://github.com/tektoncd/community/blob/master/teps/0007-conditions-beta.md).
  • Loading branch information
jerop committed Aug 24, 2020
1 parent 74cc03a commit 2ce9053
Show file tree
Hide file tree
Showing 24 changed files with 2,928 additions and 538 deletions.
53 changes: 53 additions & 0 deletions docs/pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ weight: 3
- [Using the `from` parameter](#using-the-from-parameter)
- [Using the `runAfter` parameter](#using-the-runafter-parameter)
- [Using the `retries` parameter](#using-the-retries-parameter)
- [Guard `Task` execution using `When Expressions`](#guard-task-execution-using-whenexpressions)
- [Guard `Task` execution using `Conditions`](#guard-task-execution-using-conditions)
- [Configuring the failure timeout](#configuring-the-failure-timeout)
- [Using `Results`](#using-results)
Expand Down Expand Up @@ -316,8 +317,60 @@ tasks:
name: build-push
```

### Guard `Task` execution using `WhenExpressions`

To run a `Task` only when certain conditions are met, it is possible to _guard_ task execution using
the `when` field. The `when` field allows you to list a series of references to `WhenExpressions`.

There are a lot of scenarios where `WhenExpressions` can be really useful. Some of these are:
- Checking if the name of a git branch matches
- Checking if the `Result` of a previous `Task` is as expected
- Checking if a git file has changed in the previous commits
- Checking if an image exists in the registry
- Checking if the name of a CI job matches

The components of `WhenExpressions` are `Input`, `Operator` and `Values`:

- `Input` is the input for the `WhenExpression` which can be static inputs or variables (`Parameters` or `Results`).
- `Operator` represents an `Input`'s relationship to a set of `Values`. `Operators` we will use in `WhenExpressions`
are `in` and `notin`.
- `Values` is an array of string values. The `Values` array must be non-empty. It can contain static values
or variables (`Parameters` or `Results`).

The declared `WhenExpressions` are evaluated before the `Task` is run. If all the `WhenExpressions`
evaluate to `True`, the `Task` is run. If any of the `WhenExpressions` evaluate to `False`, the `Task` is
not run and the `TaskRun` status field `ConditionSucceeded` is set to `False` with the reason set to
`WhenExpressionsEvaluatedToFalse`.

When `WhenExpressions` are specified in a `Task`, [`Conditions`](#guard-task-execution-using-conditions) should not be speficied in the same `Task`.
The `Pipeline` will be rejected as invalid if both `WhenExpressions` and `Conditions` are included.

```yaml
tasks:
- name: first-create-file
when:
- input: "$(params.path)"
operator: in
values: ["README.md"]
taskRef:
name: create-readme-file
---
tasks:
- name: echo-file-exists
when:
- input: "$(tasks.check-file.results.status)"
operator: in
values: ["exists"]
taskRef:
name: echo-file-exists
```

For an end-to-end example, see [`PipelineRun` with `WhenExpressions`](../examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml).

### Guard `Task` execution using `Conditions`

**Note:** `Conditions` are deprecated, use [`WhenExpressions`](#guard-task-execution-using-whenexpressions) instead.

To run a `Task` only when certain conditions are met, it is possible to _guard_ task execution using
the `conditions` field. The `conditions` field allows you to list a series of references to
[`Condition`](./conditions.md) resources. The declared `Conditions` are run before the `Task` is run.
Expand Down
239 changes: 239 additions & 0 deletions examples/v1beta1/pipelineruns/pipelinerun-with-when-expressions.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,239 @@
# Copied from https://github.com/tektoncd/catalog/blob/v1beta1/git/git-clone.yaml :(
# This can be deleted after we add support to refer to the remote Task in a registry (Issue #1839) or
# add support for referencing task in git directly (issue #2298)
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: git-clone-from-catalog
spec:
workspaces:
- name: output
description: The git repo will be cloned onto the volume backing this workspace
params:
- name: url
description: git url to clone
type: string
- name: revision
description: git revision to checkout (branch, tag, sha, ref…)
type: string
default: master
- name: refspec
description: (optional) git refspec to fetch before checking out revision
default: ""
- name: submodules
description: defines if the resource should initialize and fetch the submodules
type: string
default: "true"
- name: depth
description: performs a shallow clone where only the most recent commit(s) will be fetched
type: string
default: "1"
- name: sslVerify
description: defines if http.sslVerify should be set to true or false in the global git config
type: string
default: "true"
- name: subdirectory
description: subdirectory inside the "output" workspace to clone the git repo into
type: string
default: ""
- name: deleteExisting
description: clean out the contents of the repo's destination directory (if it already exists) before trying to clone the repo there
type: string
default: "false"
- name: httpProxy
description: git HTTP proxy server for non-SSL requests
type: string
default: ""
- name: httpsProxy
description: git HTTPS proxy server for SSL requests
type: string
default: ""
- name: noProxy
description: git no proxy - opt out of proxying HTTP/HTTPS requests
type: string
default: ""
results:
- name: commit
description: The precise commit SHA that was fetched by this Task
steps:
- name: clone
image: gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.12.1
script: |
CHECKOUT_DIR="$(workspaces.output.path)/$(params.subdirectory)"
cleandir() {
# Delete any existing contents of the repo directory if it exists.
#
# We don't just "rm -rf $CHECKOUT_DIR" because $CHECKOUT_DIR might be "/"
# or the root of a mounted volume.
if [[ -d "$CHECKOUT_DIR" ]] ; then
# Delete non-hidden files and directories
rm -rf "$CHECKOUT_DIR"/*
# Delete files and directories starting with . but excluding ..
rm -rf "$CHECKOUT_DIR"/.[!.]*
# Delete files and directories starting with .. plus any other character
rm -rf "$CHECKOUT_DIR"/..?*
fi
}
if [[ "$(params.deleteExisting)" == "true" ]] ; then
cleandir
fi
test -z "$(params.httpProxy)" || export HTTP_PROXY=$(params.httpProxy)
test -z "$(params.httpsProxy)" || export HTTPS_PROXY=$(params.httpsProxy)
test -z "$(params.noProxy)" || export NO_PROXY=$(params.noProxy)
/ko-app/git-init \
-url "$(params.url)" \
-revision "$(params.revision)" \
-refspec "$(params.refspec)" \
-path "$CHECKOUT_DIR" \
-sslVerify="$(params.sslVerify)" \
-submodules="$(params.submodules)" \
-depth "$(params.depth)"
cd "$CHECKOUT_DIR"
RESULT_SHA="$(git rev-parse HEAD | tr -d '\n')"
EXIT_CODE="$?"
if [ "$EXIT_CODE" != 0 ]
then
exit $EXIT_CODE
fi
# Make sure we don't add a trailing newline to the result!
echo -n "$RESULT_SHA" > $(results.commit.path)
---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: check-file
spec:
params:
- name: path
workspaces:
- name: source
description: The workspace to check for the file.
results:
- name: status
description: indicating whether the file exists
steps:
- name: check-file
image: alpine
script: |
if test -f $(workspaces.source.path)/$(params.path); then
printf exists | tee /tekton/results/status
else
printf missing | tee /tekton/results/status
fi
---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: guarded-pipeline
spec:
params:
- name: path
type: string
description: The path of the file to be created.
default: "README.md"
- name: repo-url
type: string
description: The git repository URL to clone from.
- name: branch-name
type: string
description: The git branch to clone.
workspaces:
- name: source-repo
description: |
This workspace will receive the cloned git repo and be passed
to the next Task to create a file.
tasks:
- name: fetch-repo
taskRef:
name: git-clone-from-catalog
params:
- name: url
value: $(params.repo-url)
- name: revision
value: $(params.branch-name)
workspaces:
- name: output
workspace: source-repo
- name: create-file
when:
- input: "$(params.path)"
operator: in
values: ["README.md"]
workspaces:
- name: source
workspace: source-repo
runAfter:
- fetch-repo
taskSpec:
workspaces:
- name: source
description: The workspace to create the readme file in.
steps:
- name: write-new-stuff
image: ubuntu
script: 'touch $(workspaces.source.path)/README.md'
- name: check-file
when:
- input: "foo"
operator: in
values: ["foo", "bar"]
params:
- name: path
value: "$(params.path)"
workspaces:
- name: source
workspace: source-repo
taskRef:
name: check-file
runAfter:
- create-file
- name: echo-file-exists
when:
- input: "$(tasks.check-file.results.status)"
operator: in
values: ["exists"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: 'echo file exists'
- name: task-should-be-skipped
when:
- input: "foo"
operator: notin
values: ["foo"]
- input: "foo"
operator: in
values: ["bar"]
taskSpec:
steps:
- name: echo
image: ubuntu
script: exit 1
---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
name: guarded-pr
spec:
serviceAccountName: 'default'
pipelineRef:
name: guarded-pipeline
params:
- name: repo-url
value: https://github.com/tektoncd/pipeline.git
- name: branch-name
value: master
workspaces:
- name: source-repo
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
15 changes: 15 additions & 0 deletions internal/builder/v1beta1/pipeline.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
resource "github.com/tektoncd/pipeline/pkg/apis/resource/v1alpha1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/selection"
"knative.dev/pkg/apis"
)

Expand Down Expand Up @@ -54,6 +55,9 @@ type PipelineRunStatusOp func(*v1beta1.PipelineRunStatus)
// PipelineTaskConditionOp is an operation which modifies a PipelineTaskCondition
type PipelineTaskConditionOp func(condition *v1beta1.PipelineTaskCondition)

// PipelineTaskWhenExpressionOp is an operation which modifies a WhenExpression.
type PipelineTaskWhenExpressionOp func(*v1beta1.WhenExpression)

// Pipeline creates a Pipeline with default values.
// Any number of Pipeline modifier can be passed to transform it.
func Pipeline(name string, ops ...PipelineOp) *v1beta1.Pipeline {
Expand Down Expand Up @@ -332,6 +336,17 @@ func PipelineTaskConditionResource(name, resource string, from ...string) Pipeli
}
}

// PipelineTaskWhenExpression adds a WhenExpression with the specified input, operator and values.
func PipelineTaskWhenExpression(input string, operator selection.Operator, values []string) PipelineTaskOp {
return func(pt *v1beta1.PipelineTask) {
pt.WhenExpressions = append(pt.WhenExpressions, v1beta1.WhenExpression{
Input: input,
Operator: operator,
Values: values,
})
}
}

// PipelineTaskWorkspaceBinding adds a workspace with the specified name, workspace and subpath on a PipelineTask.
func PipelineTaskWorkspaceBinding(name, workspace, subPath string) PipelineTaskOp {
return func(pt *v1beta1.PipelineTask) {
Expand Down
11 changes: 7 additions & 4 deletions internal/builder/v1beta1/pipeline_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"github.com/google/go-cmp/cmp"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/selection"
"knative.dev/pkg/apis"
duckv1beta1 "knative.dev/pkg/apis/duck/v1beta1"

Expand Down Expand Up @@ -54,6 +55,7 @@ func TestPipeline(t *testing.T) {
tb.PipelineTaskOutputResource("some-image", "my-only-image-resource"),
),
tb.PipelineTask("never-gonna", "give-you-up",
tb.PipelineTaskWhenExpression("foo", selection.In, []string{"foo", "bar"}),
tb.RunAfter("foo"),
tb.PipelineTaskTimeout(5*time.Second),
),
Expand Down Expand Up @@ -133,10 +135,11 @@ func TestPipeline(t *testing.T) {
}},
},
}, {
Name: "never-gonna",
TaskRef: &v1beta1.TaskRef{Name: "give-you-up"},
RunAfter: []string{"foo"},
Timeout: &metav1.Duration{Duration: 5 * time.Second},
Name: "never-gonna",
TaskRef: &v1beta1.TaskRef{Name: "give-you-up"},
WhenExpressions: []v1beta1.WhenExpression{{Input: "foo", Operator: selection.In, Values: []string{"foo", "bar"}}},
RunAfter: []string{"foo"},
Timeout: &metav1.Duration{Duration: 5 * time.Second},
}, {
Name: "foo",
TaskSpec: &v1beta1.EmbeddedTask{
Expand Down
Loading

0 comments on commit 2ce9053

Please sign in to comment.