Kubeflow Pipelines and Tekton

[animeshsingh]

Uber issue to track the implementation of Kubeflow Pipelines and Tekton integration. The work is currently divided into three phases, with the first phase focussed on creating a KFP Compiler for Tekton, which is getting close to ready

Design doc and specifications: http://bit.ly/kfp-tekton
Github repo: https://github.com/kubeflow/kfp-tekton
CD Foundation MLOps Sig:
https://cd.foundation/blog/2020/02/11/announcing-the-cd-foundation-mlops-sig/

In 2nd phase, we will start moving some PRs to core KFP repo to drive some changes to the code to make it easier to plug-in Tekton, as well as make the current IR more comprehensive

@paveldournov @jessiezcc @neuromage @Ark-kun @ckadner @Tomcli @jlewi

We will be using this overall issue to track the progress and prioritization

[talebzeghmi]

The design document mentions an interesting "intermediate graph representation (DAG)", has MLGraph SeldonIO/mlgraph#2 been thought of as an intermediate representation?

[jlewi]

@talebzeghmi I believe SeldonIO's mlgraph is focused on inference graphs which I think is a sufficiently different use case from the type of graphs KFP is doing.

/cc @cliveseldon

[ukclivecox]

@talebzeghmi I think I would agree that the spec of MLGraph as it stands is inference focused for k8s. The initial part has a general DAG but that is the only general part and is basically a copy of similar concepts in Argo when it was originally proposed. There is nothing particularly special about the DAG data structure.

[Tomcli]

We are looking at adding Tekton support to the KFP api-server. However, Tekton beta client required client-go v1.17.0 and this version of the go client is causing some issue with the Argo client.

GoCompile: error running compiler: exit status 2
/private/var/tmp/_bazel_tommyli/284efad53e7368dc3c6c11d0b1537251/sandbox/darwin-sandbox/91/execroot/__main__/external/com_github_argoproj_argo/pkg/client/clientset/versioned/typed/workflow/v1alpha1/workflow_client.go:58:32: undefined: serializer.DirectCodecFactory

This is also mentioned in argoproj/argo-workflows#1983 which is fixed in the newer version of Argo. Do we have any plan to migrate the Argo go client to a newer version?

[jlewi]

@Tomcli I would suggest opening a specific issue on the client go version and then adding the appropriate pipellines folks to comment on it.

[animeshsingh]

+1 on above @Tomcli. We can use this as an umbrella issue, and just point to the specific issues from here

[animeshsingh]

Another related issue being tracked for bazel update/upgrade
#3461 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[Bobgy]

/reopen
/lifecycle frozen

[k8s-ci-robot]

@Bobgy: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rimolive]

Closing this issue. Talked to Tommy and we're good to do it.

/close

[google-oss-prow]

@rimolive: Closing this issue.

In response to this:

Closing this issue. Talked to Tommy and we're good to do it.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.