Use dag execution instead of linear one

docs/pipelineruns.md

@@ -3,9 +3,8 @@
 This document defines `PipelineRuns` and their capabilities.
 
 On its own, a [`Pipeline`](pipelines.md) declares what [`Tasks`](tasks.md) to
-run, and dependencies between [`Task`](tasks.md) inputs and outputs via
-[`from`](pipelines.md#from). To execute the `Tasks` in the `Pipeline`, you must
-create a `PipelineRun`.
+run, and [the order they run in](pipelines.md#ordering). To execute the `Tasks`
+in the `Pipeline`, you must create a `PipelineRun`.
 
 Creation of a `PipelineRun` will trigger the creation of
 [`TaskRuns`](taskruns.md) for each `Task` in your pipeline.
@@ -43,15 +42,15 @@ following fields:
     object that enables your build to run with the defined authentication
     information.
   - `timeout` - Specifies timeout after which the `PipelineRun` will fail.
-  - [`nodeSelector`] - a selector which must be true for the pod to fit on a
+  - [`nodeSelector`] - A selector which must be true for the pod to fit on a
     node. The selector which must match a node's labels for the pod to be
     scheduled on that node. More info:
-    https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
-  - [`affinity`] - the pod's scheduling constraints. More info:
-    https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature
+    <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/>
+  - [`affinity`] - The pod's scheduling constraints. More info:
+    <https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#node-affinity-beta-feature>
 
 [kubernetes-overview]:
-  https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
+  <https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields>
 
 ### Resources
 

docs/pipelines.md

@@ -9,6 +9,8 @@ This document defines `Pipelines` and their capabilities.
   - [Parameters](#parameters)
   - [Pipeline Tasks](#pipeline-tasks)
     - [From](#from)
+    - [RunAfter](#runafter)
+- [Ordering](#ordering)
 - [Examples](#examples)
 
 ## Syntax
@@ -31,9 +33,15 @@ following fields:
   - [`resources`](#declared-resources) - Specifies which
     [`PipelineResources`](resources.md) of which types the `Pipeline` will be
     using in its [Tasks](#pipeline-tasks)
+  - `tasks`
+    - `resources.inputs` / `resource.outputs`
+      - [`from`](#from) - Used when the content of the [`PipelineResource`](resources.md)
+        should come from the [output](tasks.md#output) of a previous [Pipeline Task](#pipeline-tasks)
+      - [`runAfter`](#runAfter) - Used when the [Pipeline Task](#pipeline-task) should be executed
+        after another Pipeline Task, but there is no [output linking](#from) required
 
 [kubernetes-overview]:
-  https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields
+  <https://kubernetes.io/docs/concepts/overview/working-with-objects/kubernetes-objects/#required-fields>
 
 ### Declared resources
 
@@ -119,9 +127,9 @@ spec:
 
 ### Pipeline Tasks
 
-A `Pipeline` will execute a sequence of [`Tasks`](tasks.md) in the order they
-are declared in. At a minimum, this declaration must include a reference to the
-`Task`:
+A `Pipeline` will execute a graph of [`Tasks`](tasks.md) (see [ordering](#ordering)
+for how to express this graph). At a minimum, this declaration must include a
+reference to the [`Task`](tasks.md):
 
 ```yaml
 tasks:
@@ -165,16 +173,21 @@ spec:
 
 #### from
 
-Sometimes you will have `Tasks` that need to take as input the output of a
-previous `Task`, for example, an image built by a previous `Task`.
+Sometimes you will have [Pipeline Tasks](#pipeline-tasks) that need to take as
+input the output of a previous `Task`, for example, an image built by a previous `Task`.
 
-Express this dependency by adding `from` on `Resources` that your `Tasks` need.
+Express this dependency by adding `from` on [`PipelineResources`](resources.md)
+that your `Tasks` need.
 
 - The (optional) `from` key on an `input source` defines a set of previous
   `PipelineTasks` (i.e. the named instance of a `Task`) in the `Pipeline`
 - When the `from` key is specified on an input source, the version of the
   resource that is from the defined list of tasks is used
-- The name of the `PipelineResource` must correspond to a `PipelineResource`
+- `from` can support fan in and fan out
+- The `from` clause [expresses ordering](#ordering), i.e. the
+  [Pipeline Task](#pipeline-task) which provides the `PipelineResource` must run
+  _before_ the Pipeline Task which needs that `PipelineResource` as an input
+  - The name of the `PipelineResource` must correspond to a `PipelineResource`
   from the `Task` that the referenced `PipelineTask` gives as an output
 
 For example see this `Pipeline` spec:
@@ -201,6 +214,126 @@ The resource `my-image` is expected to be given to the `deploy-app` `Task` from
 the `build-app` `Task`. This means that the `PipelineResource` `my-image` must
 also be declared as an output of `build-app`.
 
+This also means that the `build-app` Pipeline Task will run before `deploy-app`,
+regardless of the order they appear in the spec.
+
+#### runAfter
+
+Sometimes you will need to have [Pipeline Tasks](#pipeline-tasks) that need to run in
+a certain order, but they do not have an explicit [output](tasks.md#outputs) to
+[input](tasks.md#inputs) dependency (which is expressed via [`from`](#from)). In this case
+you can use `runAfter` to indicate that a Pipeline Task should be run after one or more
+previous Pipeline Tasks.
+
+For example see this `Pipeline` spec:
+
+```yaml
+- name: test-app
+  taskRef:
+    name: make-test
+  resources:
+    inputs:
+      - name: my-repo
+- name: build-app
+  taskRef:
+    name: kaniko-build
+  runAfter:
+    - test-app
+  resources:
+    inputs:
+      - name: my-repo
+```
+
+In this `Pipeline`, we want to test the code before we build from it, but there is no output
+from `test-app`, so `build-app` uses `runAfter` to indicate that `test-app` should run before
+it, regardless of the order they appear in the spec.
+
+## Ordering
+
+The [Pipeline Tasks](#pipeline-tasks) in a `Pipeline` can be connected and run in a graph,
+specifically a *Directed Acyclic Graph* or DAG. Each of the Pipeline Tasks is a node, which
+can be connected (i.e. a *Graph*) such that one will run before another (i.e. *Directed*),
+and the execution will eventually complete (i.e. *Acyclic*, it will not get caught in infinite
+loops).
+
+This is done using:
+
+- [`from`](#from) clauses on the [`PipelineResources`](#resources) needed by a `Task`
+- [`runAfter`](#runAfter) clauses on the [Pipeline Tasks](#pipeline-tasks)
+
+For example see this `Pipeline` spec:
+
+```yaml
+- name: lint-repo
+  taskRef:
+    name: pylint
+  resources:
+    inputs:
+      - name: my-repo
+- name: test-app
+  taskRef:
+    name: make-test
+  resources:
+    inputs:
+      - name: my-repo
+- name: build-app
+  taskRef:
+    name: kaniko-build-app
+  runAfter:
+    - test-app
+  resources:
+    inputs:
+      - name: my-repo
+    outputs:
+      - name: image
+        resource: my-app-image
+- name: build-frontend
+  taskRef:
+    name: kaniko-build-frontend
+  runAfter:
+    - test-app
+  resources:
+    inputs:
+      - name: my-repo
+    outputs:
+      - name: image
+        resource: my-frontend-image
+- name: deploy-all
+  taskRef:
+    name: deploy-kubectl
+  resources:
+    inputs:
+      - name: my-app-image
+        from:
+          - build-app
+      - name: my-frontend-image
+        from:
+          - build-frontend
+```
+
+This will result in the following execution graph:
+
+```none
+        |            |
+        v            v
+     test-app    lint-repo
+    /        \
+   v          v
+build-app  build-frontend
+   \          /
+    v        v
+    deploy-all
+```
+
+1. The `lint-repo` and `test-app` Pipeline Tasks will begin executing simultaneously.
+   (They have no `from` or `runAfter` clauses.)
+1. Once `test-app` completes, both `build-app` and `build-frontend` will begin
+   executing simultaneously (both `runAfter` `test-app`).
+1. When both `build-app` and `build-frontend` have completed, `deploy-all` will
+   execute (it requires `PipelineResources` from both Pipeline Tasks).
+1. The entire `Pipeline` will be finished executing after `lint-repo` and `deploy-all`
+   have completed.
+
 ## Examples
 
 For complete examples, see

docs/tutorial.md

@@ -328,10 +328,12 @@ resource definition.
 
 ## Pipeline
 
-A [`Pipeline`](pipelines.md) defines a list of tasks to execute, while also
-indicating if any outputs should be used as inputs of a following task by using
-[the `from` field](pipelines.md#from). The same templating you used in tasks is
-also available in pipeline.
+A [`Pipeline`](pipelines.md) defines a list of tasks to execute in
+order, while also indicating if any outputs should be used as inputs
+of a following task by using [the `from` field](pipelines.md#from) and
+also indicating [the order of executing (using the `runAfter` and
+`from` fields)](pipelines.md#ordering). The same templating you used
+in tasks is also available in pipeline.
 
 For example:
 
@@ -612,6 +614,14 @@ annotation applies to subjects such as Docker registries, log output locations
 and other nuances that may be specific to particular cloud providers or
 services.
 
+The `TaskRuns` have been created in the following [order](pipelines.md#ordering):
+
+1. `tutorial-pipeline-run-1-build-skaffold-web` - This runs the [Pipeline Task](pipelines.md#pipeline-tasks)
+   `build-skaffold-web` first, because it has no [`from` or `runAfter` clauses](pipelines.md#ordering)
+1. `tutorial-pipeline-run-1-deploy-web` - This runs `deploy-web` second, because its [input](tasks.md#inputs)
+   `web-image` comes [`from`](pipelines.md#from) `build-skaffold-web` (therefore `build-skaffold-web`
+   must run before `deploy-web`).
+
 ---
 
 Except as otherwise noted, the content of this page is licensed under the

examples/pipelineruns/pipelinerun.yaml

@@ -50,6 +50,28 @@ spec:
 ---
 apiVersion: tekton.dev/v1alpha1
 kind: Task
+metadata:
+  name: unit-tests
+spec:
+  inputs:
+    resources:
+    - name: workspace
+      type: git
+      targetPath: go/src/github.com/GoogleContainerTools/skaffold
+  steps:
+  - name: run-tests
+    image: golang
+    env:
+    - name: GOPATH
+      value: /workspace/go
+    workingDir: /workspace/go/src/github.com/GoogleContainerTools/skaffold
+    command:
+    - make
+    args:
+    - test
+---
+apiVersion: tekton.dev/v1alpha1
+kind: Task
 metadata:
   name: build-push
 spec:
@@ -140,7 +162,15 @@ spec:
   - name: app-image
     type: image
   tasks:
+  - name: skaffold-unit-tests
+    taskRef:
+      name: unit-tests
+    resources:
+      inputs:
+      - name: workspace
+        resource: source-repo
   - name: build-skaffold-web
+    runAfter: [skaffold-unit-tests]
     taskRef:
       name: build-push
     params:
@@ -156,6 +186,7 @@ spec:
       - name: builtImage
         resource: web-image
   - name: build-skaffold-app
+    runAfter: [skaffold-unit-tests]
     taskRef:
       name: build-push
     params:

pkg/apis/pipeline/v1alpha1/pipeline_types.go

@@ -72,6 +72,12 @@ type Pipeline struct {
 type PipelineTask struct {
 	Name    string  `json:"name"`
 	TaskRef TaskRef `json:"taskRef"`
+
+	// RunAfter is the list of PipelineTask names that should be executed before
+	// this Task executes. (Used to force a specific ordering in graph execution.)
+	// +optional
+	RunAfter []string `json:"runAfter,omitempty"`
+
 	// +optional
 	Resources *PipelineTaskResources `json:"resources,omitempty"`
 	// +optional
@@ -127,6 +133,7 @@ type PipelineTaskInputResource struct {
 	// Resource is the name of the DeclaredPipelineResource to use.
 	Resource string `json:"resource"`
 	// From is the list of PipelineTask names that the resource has to come from.
+	// (Implies an ordering in the execution graph.)
 	// +optional
 	From []string `json:"from,omitempty"`
 }

pkg/apis/pipeline/v1alpha1/pipeline_validation.go

@@ -19,7 +19,7 @@ package v1alpha1
 import (
 	"fmt"
 
-	"github.com/knative/build-pipeline/pkg/reconciler/v1alpha1/taskrun/list"
+	"github.com/knative/build-pipeline/pkg/list"
 	"github.com/knative/build-pipeline/pkg/reconciler/v1alpha1/templating"
 	"github.com/knative/pkg/apis"
 	"k8s.io/apimachinery/pkg/api/equality"