Task(s) to manage kubernets resources in a cluster

[afrittoli]

Feature description

In the context of MLOps, but most likely also beyond that, it's a common use case to create / update / delete kubernetes resources as part of a Tekton pipeline.

The proposal here is to have a task or set of tasks in the catalog that would define a standard way of doing that in Tekton, which a clear and re-usable interface.

Inputs would be:

1. The resource meta, including type, name (template), namespace, labels and annotations, and the spec which could be an arbitrary JSON/YAML, which we would have to define how it is conveyed to the task
2. The target cluster, which could be a cluster PipelineResource or a set of params
3. The operation to be performed - i.e. create, replace, delete

This would be about managing single resources, so it shall not support provisioning the content of a folder or a generic YAML with multiple resources.

This could be implemented as one generic tasks that takes rather generic inputs, which could be usable for all resources, but have a weak validation - or a set of tasks, one per resource type, at least for common resources, which could have more strict checks on input params.

[vdemeester]

/kind feature

[vdemeester]

@afrittoli should this issue take over #37 ? 🙃

[afrittoli]

#37

Oh, I. missed that one :)

We, we could do that. I guess for deployments we might also have tasks that support provisioning a set of resources at once, so everything that is needed to deploy a service (including deployment, configmap, secrets, ingress etc).

[ckadner]

This could be implemented as one generic task that takes rather generic inputs, which could be usable for all resources, but have a weak validation - or a set of tasks, one per resource type, at least for common resources, which could have more strict checks on input params.

In order to streamline that implementation from a KFP-Tekton compiler perspective, we would like to follow the precedence laid out by Argo's ResourceTemplate, and create one generic task with the following parameters:

• action: ['get', 'create', 'apply', 'delete', 'replace', 'patch'] -- the action to perform to the resource
• merge_strategy: ['strategic', 'merge', 'json'] -- the strategy used to merge a patch, defaults to "strategic"
• success_condition: a label selector expression which describes the success condition
• failure_condition: a label selector expression which describes the failure condition
• manifest: the kubernetes manifest

[animeshsingh]

@vincent-pli @vdemeester @afrittoli this is a high priority requirement coming from our internal product teams. Would be good to know how can we get this prioritized? I have also discussed these with the OpenShift pipelines team, the relevance and importance of the work effort.

[vincent-pli]

Could we just use image with kubectl and use workspace to mount related yaml files to the task, then make path of the yaml file as parameter then define create, delete as args...

[ckadner]

We want to use the new "ResourceTask" via a taskRef in our pipelines and provide the manifest YAML as a string parameter without having to create files workspaces file mounts etc.

For implementing the task, I think a kubectl wrapper image would be okay. Especially from a maintenance point of view since anyone working with Kubernetes would be able to understand the commands, fix issues, try command locally, etc.

[ckadner]

A quick searching of DockerHub brings up this kubectl image that looks promising and has 10M+ pulls (Apache 2.0 license):

• https://hub.docker.com/r/bitnami/kubectl/

There are also a few recipes to cook up a custom kubectl image:

• https://itnext.io/portable-kubernetes-management-with-kubectl-in-docker-cb861a2c3c02

[vincent-pli]

@ckadner
and this one:

catalog/knctl/knctl-deploy.yaml

Line 16 in 270c833

image: gcr.io/cloud-builders/kubectl # it is huge

[Tomcli]

In Argo it was done in the controller since they already have kubectl binary in the controller image.
https://github.com/argoproj/argo/blob/master/workflow/executor/resource.go#L22-L76

[Tomcli]

After we create/delete the resource, we also need to return a status to tell whether this is up running/deleted.

[vincent-pli]

@Tomcli
I think a kubectl wrapper could satisfy you with some extra work.
for example:

• Create a pod, stuck until the pod is running:
  you need extra loop to watch the status of pod.
• Create a pod, stuck until the pod complete and make decision base on output of the pod:
  then you could mount the same pvc (from the workspace) to the new created pod...

[animeshsingh]

@vincent-pli is this something you want to take forward? Get this pushed in tekton, and we can leverage it as part of KFP?

[vincent-pli]

@animeshsingh
OK, I think I can make pr for kubectl wrapped example but no loop monitoring as @Tomcli mentioned.

[vincent-pli]

#237

[ckadner]

Thanks for your PR @vincent-pli. I just took a quick look, but I only saw one parameter resource. Ideally we would need 4 more as I described in my comment above:

#233 (comment)

• action: ['get', 'create', 'apply', 'delete', 'replace', 'patch'] -- the action to perform to the resource
• merge_strategy: ['strategic', 'merge', 'json'] -- the strategy used to merge a patch, defaults to "strategic"
• success_condition: a label selector expression which describes the success condition
• failure_condition: a label selector expression which describes the failure condition
• manifest: the kubernetes manifest

Assuming your parameter resource really is the manifest, could we add action even if initally it only supports default action create or apply?

[vincent-pli]

@ckadner

Sorry for delay, involved in a tough project.
The function you mentioned is correct and needed for a complete kubernete resource hendler features, it's too complex to just with script and yml, I think it should be a pre-prepared image.

Whatever, I will dig into and check if i can found something.

[Tomcli]

Thanks @vincent-pli, maybe you can add you kubeclient task for this? We still have some edge cases for the results that we love to have support such as kubeflow/kfp-tekton#94

[vincent-pli]

Thanks @Tomcli , I add some draft code to address your requirement, please check that:https://github.com/vincent-pli/kubectl-wrapper

[animeshsingh]

thanks folks - @vincent-pli great to see this coming in as a PR soon

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.