Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use storage bucket to link outputs and inputs of tasks #384

Closed
nader-ziada opened this issue Jan 10, 2019 · 3 comments · Fixed by #444
Closed

Use storage bucket to link outputs and inputs of tasks #384

nader-ziada opened this issue Jan 10, 2019 · 3 comments · Fixed by #444
Assignees
@nader-ziada
Labels
design This task is about creating and discussing a design meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given
Milestone
0.4 Knative Release

Comments

@nader-ziada
Copy link
Member

Expected Behavior

  • Currently the PipelineRun controller uses a PVC (Persistent volume claim) to link artifacts from the output of a task to the input of the following task.
  • Due to the nature of PVC having different implementation on different platforms, the system should also support using a cloud storage bucket, for example a gcs bucket, to achieve the same functionality.
  • The storage bucket should be configured by the user as part of the PipelineRun

Actual Behavior

Only PVCs are supported

Additional Info

related to #148
@nader-ziada
Copy link
Member Author

/assign

@bobcatfish bobcatfish added design This task is about creating and discussing a design meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given labels Jan 10, 2019
@bobcatfish bobcatfish mentioned this issue Jan 10, 2019
Closed
@bobcatfish
Copy link
Collaborator

Crazy (probably bad) idea: could it also make sense to have an option where if task X's outputs are linked to task Y's inputs, we schedule them on the same node and use an emptydir volume to share data between them?

@nader-ziada
Copy link
Member Author

I don't think you can share emptydir volume across pods

@bobcatfish bobcatfish added this to the 0.4 Knative Release milestone Jan 23, 2019
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 28, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
0918ba8 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
@nader-ziada nader-ziada mentioned this issue Jan 28, 2019
Merged
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 28, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
7007b48 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 29, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
684abbf 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 29, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
aab38d6 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 29, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
b494878 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 30, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
1a5c2be 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 30, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
5d8e812 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Jan 31, 2019
@nader-ziada
sharing artifacts between pipeline tasks using bucket
6171c41 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue tektoncd#384
knative-prow-robot pushed a commit that referenced this issue Jan 31, 2019
@nader-ziada @knative-prow-robot
sharing artifacts between pipeline tasks using bucket
985ff2f 
- using gcp buckets only for now
- bucket information configured using a config map
- refactor of pvc implementation for the same feature to use same interface
- artifact bucket and artifact pvc return the container spec to execute the upload and download steps
- issue #384
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nader-ziada nader-ziada

Labels
design This task is about creating and discussing a design meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given
Projects
None yet
Milestone
0.4 Knative Release
Development

Successfully merging a pull request may close this issue.

sharing artifacts between pipeline tasks using bucket nader-ziada/pipeline
2 participants
@bobcatfish @nader-ziada