Allow passing individual files or sets of files as Task inputs/outputs

[abayer]

Expected Behavior

A file pattern can be given as an output for a Task, with the files in the workspace matching that pattern "archived" and able to be used as an input resource for another Task.

Use case

Imagine building on Kubernetes, but not targeting Kubernetes. Say we need to build an artifact (let's go with a war file for now), and then for whatever reason, we need to run tests of that war file on different Linux distributions, but we have to use the exact same artifact we built, rather than rebuilding again. Silly, yeah, but this is a contrived example. =) We only need to copy that one file from the Task that built the war to the 1..n other Tasks that are going to run tests against that war in parallel.

Actual Behavior

• The only way to do this currently with PipelineResources would be to use GCS (which would upload and download the files between Tasks) or Git (which would check out the contents of a git repo unnecessarily)
• You could make this happen by creating a PVC yourself and making it available to both Tasks via a Volume but you wouldn't have the benefits of input and output linking

Steps to reproduce

The output-pipelinerun.yaml example is a great example of this. The whole point of this example is to write files and share them between Tasks, but a git repo is required to do this.

• The example does use the README from the repo in one of its tasks
• BUT the functionality that creates the file and then the task that reads the written file really do not need that repo at all and could be updated to use the new "file set" resource, e.g. something like this:

# Task writes "some stuff" to a predefined path in the workspace git PipelineResource
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: create-file
spec:
  inputs:
    resources:
    - name: workspace
      type: git
      targetPath: damnworkspace
  outputs:
    resources:
    - name: written-files
      type: files # this is the new bit! we dont actually need the git output now
  steps:
  - name: read-docs-old
    image: ubuntu
    command: ["/bin/bash"]
    args: ['-c', 'ls -la /workspace/damnworkspace/docs/README.md'] # tests that targetpath works
  - name: write-new-stuff
    image: ubuntu
    command: ['bash']
    args: ['-c', 'echo some stuff > /workspace/written-files/stuff']
---
# Reads a file from a predefined path in the workspace git PipelineResource
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: check-stuff-file-exists
spec:
  inputs:
    resources:
    - name: the-files
      type: files # and now we dont need git here
      targetPath: newworkspace
  steps:
  - name: read
    image: ubuntu
    command: ["/bin/bash"]
    args: ['-c', 'cat /workspace/newworkspace/stuff'] # tests that new targetpath and previous task output is dumped

Additional Info

Depends on #255

[bobcatfish]

@abayer would you expect any files in the container to be fair game, or only those that are mounted at /workspace as resources?

Is it possible to add a use case to demonstrate when we'd see this in action?

[abayer]

I’d say only those under /workspace would be reasonable. Will get the use case up tomorrow!

[abayer]

And a use case:

Imagine building on Kubernetes, but not targeting Kubernetes. Say we need to build an artifact (let's go with a war file for now), and then for whatever reason, we need to run tests of that war file on different Linux distributions, but we have to use the exact same artifact we built, rather than rebuilding again. Silly, yeah, but this is a contrived example. =) We only need to copy that one file from the Task that built the war to the 1..n other Tasks that are going to run tests against that war in parallel.

That make sense?

[bobcatfish]

Makes sense @abayer ! I updated the description with an example, feel free to fix it if this doesn't reflect what you're looking for.

[bobcatfish]

The file set resource (#1285 ) might address this?

[bobcatfish]

#1285 + #1184 will cover this ! 🎉