Workflow of Workflows example does not work on V3.1-rc5

[amitm02]

Summary

Problem running workflow of workflows (https://argoproj.github.io/argo-workflows/workflow-of-workflows/).
Following the instructions, parent workflow fails with

time="2021-05-24T10:47:20.969Z" level=info msg="Starting Workflow Executor" executorType=emissary version=v3.1.0-rc5
time="2021-05-24T10:47:20.973Z" level=info msg="Executor (version: v3.1.0-rc5, build_date: 2021-05-17T21:32:14Z) initialized (pod: argo/workflow-of-workflows-bf4m5-3399545254) with template:\n{\"name\":\"triggerWorkflowUsingResourceWithoutArgument\",\"inputs\":{\"parameters\":[{\"name\":\"workflowtemplate\",\"value\":\"workflow-template-submittable\"}]},\"outputs\":{},\"metadata\":{},\"resource\":{\"action\":\"create\",\"manifest\":\"apiVersion: argoproj.io/v1alpha1\\nkind: Workflow\\nmetadata:\\n  generateName: workflow-of-workflows-1-\\nspec:\\n  workflowTemplateRef:\\n    name: workflow-template-submittable\\n\",\"successCondition\":\"status.phase == Succeeded\",\"failureCondition\":\"status.phase in (Failed, Error)\"},\"archiveLocation\":{\"archiveLogs\":true,\"s3\":{\"endpoint\":\"minio:9000\",\"bucket\":\"my-bucket\",\"insecure\":true,\"accessKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"accesskey\"},\"secretKeySecret\":{\"name\":\"my-minio-cred\",\"key\":\"secretkey\"},\"key\":\"workflow-of-workflows-bf4m5/workflow-of-workflows-bf4m5-3399545254\"}}}"
time="2021-05-24T10:47:20.973Z" level=info msg="Loading manifest to /tmp/manifest.yaml"
time="2021-05-24T10:47:20.973Z" level=info msg="kubectl create -f /tmp/manifest.yaml -o json"
time="2021-05-24T10:47:21.973Z" level=info msg="Resource: argo/Workflow.argoproj.io/workflow-of-workflows-1-wlkwg. SelfLink: /apis/argoproj.io/v1alpha1/namespaces/argo/workflows/workflow-of-workflows-1-wlkwg"
time="2021-05-24T10:47:21.973Z" level=info msg="Starting SIGUSR2 signal monitor"
time="2021-05-24T10:47:21.973Z" level=info msg="Waiting for conditions: status.phase==Succeeded"
time="2021-05-24T10:47:21.973Z" level=info msg="Failing for conditions: status.phase in (Error,Failed)"
time="2021-05-24T10:47:21.980Z" level=info msg="Get workflows 200"
time="2021-05-24T10:47:21.980Z" level=info msg="{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Workflow\",\"metadata\":{\"creationTimestamp\":\"2021-05-24T10:47:21Z\",\"generateName\":\"workflow-of-workflows-1-\",\"generation\":1,\"managedFields\":[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:generateName\":{}},\"f:spec\":{\".\":{},\"f:workflowTemplateRef\":{}}},\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"time\":\"2021-05-24T10:47:21Z\"}],\"name\":\"workflow-of-workflows-1-wlkwg\",\"namespace\":\"argo\",\"resourceVersion\":\"30897600\",\"selfLink\":\"/apis/argoproj.io/v1alpha1/namespaces/argo/workflows/workflow-of-workflows-1-wlkwg\",\"uid\":\"6e03bdab-8b88-4e06-b898-d9f8bd0c4831\"},\"spec\":{\"workflowTemplateRef\":{\"name\":\"workflow-template-submittable\"}}}\n"
time="2021-05-24T10:47:21.980Z" level=info msg="failure condition '{status.phase in [Error Failed]}' evaluated false"
time="2021-05-24T10:47:21.980Z" level=info msg="success condition '{status.phase == [Succeeded]}' evaluated false"
time="2021-05-24T10:47:21.980Z" level=info msg="0/1 success conditions matched"
time="2021-05-24T10:47:21.980Z" level=info msg="Waiting for resource Workflow.argoproj.io/workflow-of-workflows-1-wlkwg in namespace argo resulted in retryable error: Neither success condition nor the failure condition has been matched. Retrying..."
time="2021-05-24T10:47:26.984Z" level=info msg="Get workflows 200"
time="2021-05-24T10:47:26.984Z" level=info msg="{\"apiVersion\":\"argoproj.io/v1alpha1\",\"kind\":\"Workflow\",\"metadata\":{\"creationTimestamp\":\"2021-05-24T10:47:21Z\",\"generateName\":\"workflow-of-workflows-1-\",\"generation\":2,\"labels\":{\"workflows.argoproj.io/phase\":\"Running\"},\"managedFields\":[{\"apiVersion\":\"argoproj.io/v1alpha1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:generateName\":{}},\"f:spec\":{\".\":{},\"f:workflowTemplateRef\":{}}},\"manager\":\"kubectl-create\",\"operation\":\"Update\",\"time\":\"2021-05-24T10:47:21Z\"},{\"apiVersion\":\"argoproj.io/v1alpha1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:labels\":{\".\":{},\"f:workflows.argoproj.io/phase\":{}}},\"f:spec\":{\"f:arguments\":{}},\"f:status\":{\".\":{},\"f:artifactRepositoryRef\":{},\"f:finishedAt\":{},\"f:nodes\":{},\"f:phase\":{},\"f:progress\":{},\"f:startedAt\":{},\"f:storedTemplates\":{},\"f:storedWorkflowTemplateSpec\":{}}},\"manager\":\"workflow-controller\",\"operation\":\"Update\",\"time\":\"2021-05-24T10:47:21Z\"}],\"name\":\"workflow-of-workflows-1-wlkwg\",\"namespace\":\"argo\",\"resourceVersion\":\"30897605\",\"selfLink\":\"/apis/argoproj.io/v1alpha1/namespaces/argo/workflows/workflow-of-workflows-1-wlkwg\",\"uid\":\"6e03bdab-8b88-4e06-b898-d9f8bd0c4831\"},\"spec\":{\"arguments\":{},\"workflowTemplateRef\":{\"name\":\"workflow-template-submittable\"}},\"status\":{\"artifactRepositoryRef\":{\"configMap\":\"artifact-repositories\",\"key\":\"default-v1\",\"namespace\":\"argo\"},\"finishedAt\":null,\"nodes\":{\"workflow-of-workflows-1-wlkwg\":{\"displayName\":\"workflow-of-workflows-1-wlkwg\",\"finishedAt\":null,\"id\":\"workflow-of-workflows-1-wlkwg\",\"inputs\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello world\"}]},\"name\":\"workflow-of-workflows-1-wlkwg\",\"phase\":\"Pending\",\"progress\":\"0/1\",\"startedAt\":\"2021-05-24T10:47:21Z\",\"templateName\":\"whalesay-template\",\"templateScope\":\"local/\",\"type\":\"Pod\"}},\"phase\":\"Running\",\"progress\":\"0/1\",\"startedAt\":\"2021-05-24T10:47:21Z\",\"storedTemplates\":{\"namespaced/workflow-template-submittable/whalesay-template\":{\"container\":{\"args\":[\"{{inputs.parameters.message}}\"],\"command\":[\"cowsay\"],\"image\":\"docker/whalesay\",\"name\":\"\",\"resources\":{}},\"inputs\":{\"parameters\":[{\"name\":\"message\"}]},\"metadata\":{},\"name\":\"whalesay-template\",\"outputs\":{}}},\"storedWorkflowTemplateSpec\":{\"arguments\":{\"parameters\":[{\"name\":\"message\",\"value\":\"hello world\"}]},\"entrypoint\":\"whalesay-template\",\"templates\":[{\"container\":{\"args\":[\"{{inputs.parameters.message}}\"],\"command\":[\"cowsay\"],\"image\":\"docker/whalesay\",\"name\":\"\",\"resources\":{}},\"inputs\":{\"parameters\":[{\"name\":\"message\"}]},\"metadata\":{},\"name\":\"whalesay-template\",\"outputs\":{}}],\"workflowTemplateRef\":{\"name\":\"workflow-template-submittable\"}}}}\n"
time="2021-05-24T10:47:26.985Z" level=info msg="failure condition '{status.phase in [Error Failed]}' evaluated false"
time="2021-05-24T10:47:26.985Z" level=info msg="success condition '{status.phase == [Succeeded]}' evaluated false"
time="2021-05-24T10:47:26.985Z" level=info msg="0/1 success conditions matched"
time="2021-05-24T10:47:26.985Z" level=info msg="Waiting for resource Workflow.argoproj.io/workflow-of-workflows-1-wlkwg in namespace argo resulted in retryable error: Neither success condition nor the failure condition has been matched. Retrying..."
Error: signal: terminated

Diagnostics

What Kubernetes provider are you using?
GKE

What version of Argo Workflows are you running?
V3.1-rc5

What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Emissary

child-wf.yaml.txt
parent-wf.yaml.txt

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[terrytangyuan]

I think this is because executor container got killed and probably can be fixed by #6033.

Could you also paste the controller pod's log?