Reorganize Catalog based on Catalog Organization proposal

[PuneetPunamiya]

Reorganize Catalog based on Catalog Organization proposal

Expected Behavior

1. Catalog Organization proposal

2. Each resource should follow the following structure

    ./task/         👈 the kind of the resource 
      /argocd       👈 definition file must have same name
        /OWNERS     👈 owners of this resource
        /README.md  👈 fallback README if version is missing README
        /0.1
          /README.md     👈 [optional] since there can be a fallback
          /argocd.yaml   👈 the file name must be resource name
          /samples/deploy-to-k8s.yaml
        /0.2/...
      /golang-build
        /0.1
          /README.md
          /golang-build.yaml
          /samples/golang-build.yaml

3. Resource YAML file includes following changes

• Labels include the version of the resource.
• Annotations include minimum pipeline version supported by the resource,
  tags associated with the resource and displayName of the resource

  
labels:
  app.kubernetes.io/version: "0.1"            👈 version of the resource
  
annotations:
  tekton.dev/pipelines.minVersion: "0.12.1"   👈 will work from x version  of pipeline
  tags: "ansible, cli"                        👈 Comma separated list of tags 
  displayName: "Ansible Tower Cli"    
  
spec:
description: |-
  ansible-tower-cli task simplifies 
  workflow, jobs, manage users...   👈 # Summary
  
  Ansible Tower (formerly ‘AWX’) is a ...

4. API breakage is prevented because there will be a new version

Actual Behavior

1. Versions are not supported by the resources
2. Minimum pipeline version supported by the resource is not specified

[PuneetPunamiya]

/assign

[necccc]

Is it possible to gain control on task versions from a pipeline?

Example scenario:

I have a Pipeline resource I wish to distribute.

With the current proposal there are some constraints I have to face:

• if I would like to use a new version of a task in the Pipeline, after that change I need to communicate that every user who uses the Pipeline should update the definitions they pull the task from
• this kinda feels like a breaking change, even if the new task itself does not introduce a breaking change

If I could set the task version on a pipeline level, this would not cause problems,
only those changes need maintenance from Pipeline users, which are truly breaking.

Ideal setup would be like:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: task-from-catalog
      taskRef:
        name: task-alpha
        version: "1.0.1"
    - name: task-from-another-catalog
      taskRef:
        name: task-beta
        version: "2.1.1"

Since k8s resources of the same kind should have unique names, currently this could be implemented in putting the version to the task name.

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: task-from-catalog
      taskRef:
        name: task-alpha-1.0.1
    - name: task-from-another-catalog
      taskRef:
        name: task-beta-2.1.1

With a similar file structure like the proposal

./task/
      /task-alpha
        /1.0.0.yaml
        /1.0.1.yaml
        ...
      /task-beta
        /2.1.0.yaml
        /2.1.1.yaml

What do you think? Is this an issue?
How will the current solution fit in with pipelineTasks later?
Or does the Hub resolves this?

[vdemeester]

What do you think? Is this an issue?
How will the current solution fit in with pipelineTasks later?
Or does the Hub resolves this?

So far this is "on the user" to do that, aka including some versionning in their Tasks. So in a gist, this is still an issue — or more accurately this is something we need to work on and fix at some point. There is a bunch of possibilities to explore here:

• Have the catalog include the version in the name field of the Tasks. That's the easiest way to fix that problem initially as it doesn't require any change on the API side.

• Have the hub do that instead — aka mutate live the task the user wants to install so that the name contains the version. This complexify the hub but same as above, this doesn't require any API change.

• There is some work that would help fix that : Tekton OCI Bundle should help there. By packaging tekton resources (task, pipelines, …) into oci artifact (images), we can version them. If we then update the TaskRef/PipelineRef fields to reference to those images (instead of an existing resource in the cluster), the versionning problem is fixed (and you can even use digest there to make sure the image didn't overwrite your tag). For more detail than the TEP, there is some prior design docs around this idea.

  • Tekton OCI Image Catalog
  • Tekton OCI Image Design

  An example of what it could look like:

  apiVersion: tekton.dev/v1alpha1
  kind: TaskRun
  metadata:
    name: my-task-run
  spec:
    taskRef:
      image:
        name: gcr.io/my/catalog:v1.2.3
        task: my-task

• Going with the above point, we could also allow referencing task directly from an HTTP endpoint, and thus allow to use the versionned URL of a task (ram.github…).

And of course, we are definitely welcoming any feedback or ideas around this (or anything else really) 😉

/cc @sthaha @bobcatfish @afrittoli @chmouel

[sthaha]

@necccc Thank for bringing up the concerns, I will try and answer as much as I can

Is it possible to gain control on task versions from a pipeline?

Example scenario:

I have a Pipeline resource I wish to distribute.

With the current proposal there are some constraints I have to face:

• if I would like to use a new version of a task in the Pipeline, after that change
  I need to communicate that every user who uses the Pipeline should update
  the definitions they pull the task from

What you are alluding to is an issue we already have with the existing system.
I think the problem is "how do I specify what the dependencies of the current
resource is". We had a discussion on this in our Catalog & Hub WG meeting
and I plan to write a TEP where we can discuss how to resolve this issue. In
broad terms I am thinking of

1. Allow annotations to resources itself but I find it less flexible

metadata:
  labels:
    version: 1.0 -> 1.1
  annotations:
     tekton.dev/dependency-01: official:task:foobar:1.0
     tekton.dev/dependency-02: official:task:bar:1.0
     tekton.dev/dependency-02: official:condition:bar:1.0
     tekton.dev/dependency-03: community:task:bar:1.0

2. A way to package Tekton and required k8s resources like the helm chart

# path: /bundle/openwhisk/1.0

apiVersion: tekton.dev/v1alpha1
kind: Bundle
metadata:
  annotations:
  labels:
    io.kubernetes/version: "1.0"

spec:
   catalogs:
      - name: foobar, url: http/my/catalog

  tasks:
    - url: http://github.com/tektoncd/catalog/task/foobar/1.0/foobar.yaml

    - catalog: community name: foobar version: 1.0
    - catalog: foobar    name: foobar version: 1.0

    - name: bar
      version: 2.0

  conditions:
    - name: openwhisk, version: 1.8

  pipelines:
    - name: openwhisk
      version: 1.0
    - name: openwhisk-knative
      version: 2.4

• this kinda feels like a breaking change, even if the new task itself does not introduce a breaking change

If the task doesn't really change its behaviour i.e. the tests of the existing
version can be run against the new submission, then I think we have a consensus
that a new version need not be created

cc: @vdemeester @afrittoli

If I could set the task version on a pipeline level, this would not cause problems,
only those changes need maintenance from Pipeline users, which are truly breaking.

Ideal setup would be like:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: task-from-catalog
      taskRef:
        name: task-alpha
        version: "1.0.1"
    - name: task-from-another-catalog
      taskRef:
        name: task-beta
        version: "2.1.1"

We should dicuss this further but I think the OCI artifact (tekton bundle)
already solves this (e.g. see below). I am not sure if the Pipeline controller
which I think is a lower layer component should be aware of a higher level
concept - catalog.

---
# ...
spec:
  tasks:
    - name: task-from-catalog
      taskRef:
        name: task-alpha
        image: task-alpha:1.0.1

    - name: task-from-another-catalog
      taskRef:
        name: task-beta
        version: "2.1.1"

Perhaps we should see if a task can be directly resolved by its URL. Thus
allowing symlinks to latest version of a task. Perhaps something like

/task/foobar/1.0
/task/foobar/1.2
/task/foobar/1.3 << --- can be referred as http://catalog/task/foobar/1/foobar.yaml
/task/foobar/2.0
/task/foobar/2.1 << --- can be referred as http://catalog/task/foobar/latest/foobar.yaml
                    `-- can be referred as http://catalog/task/foobar/2/foobar.yaml

Since k8s resources of the same kind should have unique names, currently this could be implemented in putting the version to the task name.

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pipeline
spec:
  tasks:
    - name: task-from-catalog
      taskRef:
        name: task-alpha-1.0.1
    - name: task-from-another-catalog
      taskRef:
        name: task-beta-2.1.1

IMHO encoding version information in the name itself isn't a good idea unless
the delimiter can't part of a valid taskname

With a similar file structure like the proposal

./task/
      /task-alpha
        /1.0.0.yaml
        /1.0.1.yaml
        ...
      /task-beta
        /2.1.0.yaml
        /2.1.1.yaml

I had considered this while writing the proposal but when you also also
consider supporting files for each verions - e.g. tests, README.md, supporting-files etc, we will have to invariably endup in creating one
additional directory to indicate the version. The other advantage of using
directory and opposed to encoding version information in file name is again
the same that we won't be able to demarkate name from version.

What do you think? Is this an issue?

With every design there will be tradeoffs, if we can define the problems in the
design, we may be able to addres it better. I have tried to provide some ideas
around issues you raised. Perhaps we can discuss this in our WG meeting

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[necccc]

Thx @sthaha for the great answer!
I'll dive into the OCI artifacts a bit more, and keep an eye on TEPs regarding these concerns.