Ability to manipulate other K8s resources as part of workflows

[jessesuen]

This has come up several times now, in that it is highly desired to create/update/delete/patch other types of kubernetes resources as part of a workflow, including other custom resources that argo does not even know about. This will be important as CRDs become more and more common.

1. resource templates
   One approach to solving this is to introduce the concept of a "resource" template type, which allows inlining other manifests into an argo workflow manifest. This template type would have a free-form manifest field that would be implemented as an golang interface{} and would be allowed to have any value. We could do some basic validation to ensure it looks like a k8s resource (or validate the core ones which are part of kubernetes), but aside from that, it would be free-form. The following is an proposal of example "resource" template which would apply an argo ui service.

spec:
  templates:
  - name: argo-ui-service
    resource:
      action: apply
      manifest:
        apiVersion: v1
        kind: Service
        metadata:
        labels:
          app: argo-ui
        name: argo-ui
        spec:
        ports:
        - port: 80
          targetPort: 8001
        selector:
          app: argo-ui
        type: LoadBalancer

A "resource" template type would simply act as syntactic sugar for a kubectl create|apply|delete|patch -f <templatename>. This approach makes it possible to orchestrate a flow of any k8s resource, while benefiting from argo workflow features such as parameter substitution.

One pitfall is that inlining manifests inside our workflow is a poor way of organizing many manifests, since most people would want to keep these as separate files. The workflow manifest itself could grow and become unmanageable for users. This is potentially solvable by adding support for referencing filepaths, or higher level tooling

2. lambda/callback templates
   Another approach is to provide very generic, general purpose lambda/callback mechanism for a workflow step. A lambda template would sit and do nothing but wait for a callback before proceeding. This completely opens up the ability for a user to define when to continue along a workflow (including creating a k8s resource and waiting for it to be ready). Another application of lambda/callback steps would be for approval steps.

[jessesuen]

Commit 25be5fd adds support for creating and waiting against any k8s resource.