feat!: use argo emissary executor by default. Fixes #5718

[Bobgy]

Description of your changes:
Fixes #5718

Breaking change

Argo emissary executor requires that each container has explicit command and args defined, refer to https://argoproj.github.io/argo-workflows/workflow-executors/#emissary-emissary.

Why we decide to adopt the breaking change? We'd have the same breaking change in bit.ly/kfp-v2-compatible and bit.ly/kfp-v2. So I think it aligns with KFP's long term goal, and it's OK to introduce the breaking change now.

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[Bobgy]

/retest

[Bobgy]

/retest

[Bobgy]

/retest

[Bobgy]

/retest

[Bobgy]

Current blocker:

the non-root pod's main container successfully executed and finished, but the wait container did not terminate.

2021-07-15 15:39:08.000 HKT main container
+ exit 0
2021-07-15 15:42:31.136 HKT wait container
time="2021-07-15T07:42:31.135Z" level=info msg="Alloc=4053 TotalAlloc=9145 Sys=73809 NumGC=5 Goroutines=8"
2021-07-15 15:47:31.136 HKT wait container
time="2021-07-15T07:47:31.135Z" level=info msg="Alloc=4075 TotalAlloc=9253 Sys=73809 NumGC=7 Goroutines=8"

https://storage.googleapis.com/oss-prow/pr-logs/pull/kubeflow_pipelines/5926/kubeflow-pipeline-e2e-test/1415563732758564864/artifacts/pods_info/integration-test-vwp75-1409706194.txt

[Bobgy]

/retest

[Bobgy]

The remaining problem is that the sidecar.py sample is not ending.
I found logs from the sidecar container:

2021/07/21 11:30:04 [ERR] Unknown signal terminated
2021/07/21 11:30:04 [ERR] Unknown signal urgent I/O condition
2021/07/21 11:30:08 [ERR] Unknown signal urgent I/O condition

https://storage.googleapis.com/oss-prow/pr-logs/pull/kubeflow_pipelines/5926/kubeflow-pipeline-sample-test/1417793963884023808/artifacts/pods_info/pipeline-with-sidecar-bl64j-3037665903.txt

[Bobgy]

/retest

[Bobgy]

/test kubeflow-pipeline-mkp-test

[Bobgy]

/approve

[Bobgy]

Merging early to prepare for RC release

[Bobgy]

/approve

[google-oss-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• manifests/gcp_marketplace/OWNERS [Bobgy]
• manifests/kustomize/OWNERS [Bobgy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Ark-kun]

I think we should start warning the users that they need to always set command: #4800

[Bobgy]

After more discussion, because emissary is still Alpha in argo.
I'll propose to change it to the default in a later release and ask for more feedback for trying it out in the next KFP community meeting.