Make low level APIs for DSL ContainerOp by leveraging k8s and argo library

[hongye-sun]

Currently, ContainerOp is a wrapper of Argo's template object which also wraps a k8s container object. The current high level APIs are not flexible enough that user can edit the full spec of the wrapped object.

It's better to provide a low level API to let advanced user to use Argo and k8s py lib to build the underlying object directly, while still keep high level APIs easy to use for most of the users.

[jlewi]

Should there be a clean separation between the APIs and the python library?
Should the primary goal of python library to be able to construct API objects using idiomatic python?

A model would be something like OpenAPI where the API can be defined in a language independent manner. For a given API, one can then create client libraries in a given language to construct it an idiomatic fashion.

Related question, how can the DSL be easily extended to orchestrate custom Kubernetes resources? It would be nice if I could just "inline" custom resources (similar to what Argo supports argoproj/argo-workflows#606).

I think the desired semantics would be something like

1. Create a custom resource
2. Wait for the specified conditions to be reached

[qimingj]

Is there a strong reason that DSL needs to follow the OpenAPI pattern? I don't foresee in the near future another DSL with a different programming language. Introducing a language neutral API is more useful when you want to expose it in multiple languages.

CRD support --- yes. I would like to support two levels of K8sOp. The low level is arbitrary resource:

op = K8sOp(resource_spec, jsonPath='...')

We should also provide resource specific high level ops:

op = TfJobOp(container_image='...', ...)

One gap is that argo only supports gathering output from job status, and not all CRDs insert results there. This requirement is important because we want to pass output to downstream components.

[hongye-sun]

There are 3 asks in this issue:

1. Provide a way for dsl user to set argo and k8s spec directly.
2. Clear separation of language independent model and python library.
3. Support argo resource template in DSL

1 and 2 are highly coupled. 3 is a little irrelevant to the topic, so I created another issue #429 to track it.

Currently, dsl is the idiomatic python lib which can directly be compiled into argo yaml. It makes sense to me to add another layer to have dsl to compile to a argo spec object, which is language independent, and provide a way for user to modify it before output to disk.

For example:
op = ContainerOp(...) op.addTransform(lambda argoTemplate: ( argoTemplate.container().addVolume(...), argoTemplate.addNodeSelector(...)))

[vicaire]

@hongye-sun, can we close this now?

[hongye-sun]

Yes, this issue has been addressed by #801 .