Support endpoint,bucket fields for artifact access in runtime workflow status

[zijianjoy]

Summary

In this PR https://github.com/argoproj/argo-workflows/pull/3377/files#diff-e513d8bd9786d929b66439f58e6a961bc77e4971998657573fbb047fe2e399d8, S3Artifact is removed because they are not being used. However, UI is unable to get bucket name from runtime workflow_manifest, therefore UI cannot load artifact preview using workflow_manifest. Suggest to include bucket and endpoint fields to status, which are needed for fetching artifact content from UI. See example:

status:
  artifactRepositoryRef:
    default: true
    bucket: 'mlpipeline'
    endpoint: 'minio-service.kubeflow:9000'

Use Cases

Kubeflow Pipelines recently upgraded argo from 2.12.9 to 3.1.0. During the upgrade, one of our existing features are broken because of fields changes in interface artifact. Previously we were able to construct request to fetch file from storage based on runtime workflow_manifest: https://github.com/kubeflow/pipelines/blob/120da92d0c07f2a4b3abdd61377dbbeb2085571a/frontend/src/components/ArtifactPreview.tsx#L121-L147. workflow_manifest was self-contained to describe the workflow information, however, new argo template only contains key: refer to example kubeflow/pipelines#5930 (comment).

Reference

kubeflow/pipelines#5944
kubeflow/pipelines#5930

cc @Bobgy

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[terrytangyuan]

The fields were removed since they were not being used at all in the codebase and these changes have been there for a year since v2.10. I am not sure if it's a good idea to rely on those fields.

[zijianjoy]

@terrytangyuan Hello Terry, instead of bringing these fields back to the old format, can we support providing the content of bucket and endpoint as part of configMap in ArtifactRepositoryRef?

Please see the status->ArtifactRepositoryRef in kubeflow/pipelines#5930 (comment): If we have bucket and endpoint information here, it will be beneficial to provide file download and preview ability.

Reference:
https://github.com/argoproj/argo-workflows/blob/4e450e250168e6b4d51a126b784e90b11a0162bc/docs/artifact-repository-ref.md

[Bobgy]

The main information KFP wants is to parse workflow status and understand artifact source, URI, endpoint, secrets info.

I agree it's not a good idea to duplicate them per artifact.
However, IIUC right now the other information can only be found in argo runtime env. What if someone changes some config? How do we know the original artifact repository a workflow ran as.
Therefore this feature request is to ask argo to persist the resolved artifact repository info to workflow status (as a separate field)

[Bobgy]

Hi @alexec @sarabala1979, this is a blocker for Kubeflow Pipelines to upgrade to argo v3.1.x.
What do you think about it?

[terrytangyuan]

Sounds good to me but I'll defer to others to decide whether to add those additional fields.

[alexec]

One of the secondary benefits of removing this information is the reduced size of NodeStatus. I'm not keen to loose that because (a) big workflows are a performance issue and (b) that data can be found in the namespace:

https://github.com/argoproj/argo-workflows/blob/master/server/artifacts/artifact_server.go#L157

[Bobgy]

Hi @alexec, for clarification, we propose to add the information back to workflow status, rather than node status, so size is not a concern as each workflow only records the status once.

[alexec]

That seems sensible. Would you like to raise a PR?

[alexec]

Actually, hold-off from a PR, I can see a way to fix this AND use it to fix out a performance issue at the same time.

[alexec]

I've just pushed dev-6255. Can you please try argoproj/workflow-controller:dev-6255 and see if that is what you want?

[Bobgy]

Awesome, will have a try!

[zijianjoy]

Thank you @alexec ! I tried the dev image and encountered an issue, is there anything we are missing?

Steps

1. In our workflow-controller deployment, replace the image field with docker.io/argoproj/workflow-controller:dev-6255.
2. Use the example workflow in https://argoproj.github.io/argo-workflows/examples/#dag.
3. Encounter the following error: 'Error (exit code 2): open /argo/podmetadata/annotations: no such file or directory'

Full dag status yaml:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"annotations":{},"name":"dag-diamond-1","namespace":"kubeflow","serviceAccountName":"pipeline-runner"},"spec":{"entrypoint":"diamond","templates":[{"container":{"command":["echo","{{inputs.parameters.message}}"],"image":"alpine:3.7"},"inputs":{"parameters":[{"name":"message"}]},"name":"echo"},{"dag":{"tasks":[{"arguments":{"parameters":[{"name":"message","value":"A"}]},"name":"A","template":"echo"},{"arguments":{"parameters":[{"name":"message","value":"B"}]},"dependencies":["A"],"name":"B","template":"echo"},{"arguments":{"parameters":[{"name":"message","value":"C"}]},"dependencies":["A"],"name":"C","template":"echo"},{"arguments":{"parameters":[{"name":"message","value":"D"}]},"dependencies":["B","C"],"name":"D","template":"echo"}]},"name":"diamond"}]}}
  creationTimestamp: "2021-07-08T04:58:59Z"
  generation: 3
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Error
    manager: workflow-controller
    operation: Update
    time: "2021-07-08T04:59:09Z"
  name: dag-diamond-1
  namespace: kubeflow
  resourceVersion: "25286133"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/kubeflow/workflows/dag-diamond-1
  uid: 9b3d593c-4195-4f0a-ba49-a0f978ca3447
spec:
  arguments: {}
  entrypoint: diamond
  templates:
  - container:
      command:
      - echo
      - '{{inputs.parameters.message}}'
      image: alpine:3.7
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: message
    metadata: {}
    name: echo
    outputs: {}
  - dag:
      tasks:
      - arguments:
          parameters:
          - name: message
            value: A
        name: A
        template: echo
      - arguments:
          parameters:
          - name: message
            value: B
        dependencies:
        - A
        name: B
        template: echo
      - arguments:
          parameters:
          - name: message
            value: C
        dependencies:
        - A
        name: C
        template: echo
      - arguments:
          parameters:
          - name: message
            value: D
        dependencies:
        - B
        - C
        name: D
        template: echo
    inputs: {}
    metadata: {}
    name: diamond
    outputs: {}
status:
  artifactRepositoryRef:
    artifactRepository:
      archiveLogs: true
      s3:
        accessKeySecret:
          key: accesskey
          name: mlpipeline-minio-artifact
        bucket: mlpipeline
        endpoint: minio-service.kubeflow:9000
        insecure: true
        keyFormat: artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}
        secretKeySecret:
          key: secretkey
          name: mlpipeline-minio-artifact
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2021-07-08T04:59:09Z"
  nodes:
    dag-diamond-1:
      children:
      - dag-diamond-1-4281134405
      displayName: dag-diamond-1
      finishedAt: "2021-07-08T04:59:09Z"
      id: dag-diamond-1
      name: dag-diamond-1
      outboundNodes:
      - dag-diamond-1-4197246310
      phase: Error
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 0
      startedAt: "2021-07-08T04:58:59Z"
      templateName: diamond
      templateScope: local/dag-diamond-1
      type: DAG
    dag-diamond-1-4197246310:
      boundaryID: dag-diamond-1
      displayName: D
      finishedAt: "2021-07-08T04:59:09Z"
      id: dag-diamond-1-4197246310
      message: 'omitted: depends condition not met'
      name: dag-diamond-1.D
      phase: Omitted
      startedAt: "2021-07-08T04:59:09Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Skipped
    dag-diamond-1-4230801548:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4197246310
      displayName: B
      finishedAt: "2021-07-08T04:59:09Z"
      id: dag-diamond-1-4230801548
      message: 'omitted: depends condition not met'
      name: dag-diamond-1.B
      phase: Omitted
      startedAt: "2021-07-08T04:59:09Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Skipped
    dag-diamond-1-4247579167:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4197246310
      displayName: C
      finishedAt: "2021-07-08T04:59:09Z"
      id: dag-diamond-1-4247579167
      message: 'omitted: depends condition not met'
      name: dag-diamond-1.C
      phase: Omitted
      startedAt: "2021-07-08T04:59:09Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Skipped
    dag-diamond-1-4281134405:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4230801548
      - dag-diamond-1-4247579167
      displayName: A
      finishedAt: "2021-07-08T04:59:02Z"
      id: dag-diamond-1-4281134405
      inputs:
        parameters:
        - name: message
          value: A
      message: 'Error (exit code 2): open /argo/podmetadata/annotations: no such file
        or directory'
      name: dag-diamond-1.A
      outputs:
        exitCode: "0"
      phase: Error
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 0
      startedAt: "2021-07-08T04:58:59Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod
  phase: Error
  progress: 1/1
  resourcesDuration:
    cpu: 1
    memory: 0
  startedAt: "2021-07-08T04:58:59Z"

[terrytangyuan]

Did you also update --executor-image like in https://github.com/argoproj/argo-workflows/blob/master/manifests/namespace-install.yaml#L494 to docker.io/argoproj/workflow-controller:dev-6255?

[alexec]

@terrytangyuan is correct. I didn’t think you need to update the executor image, but you’ll need to update that too.

[zijianjoy]

Thank you @terrytangyuan and @alexec !

I have updated the executor-image flag to the dev image too. This time the error has changed to the following:

dag-diamond-1-4281134405:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4230801548
      - dag-diamond-1-4247579167
      displayName: A
      finishedAt: "2021-07-08T16:25:55Z"
      id: dag-diamond-1-4281134405
      inputs:
        parameters:
        - name: message
          value: A
      message: 'ContainerCannotRun (exit code 127): OCI runtime create failed: container_linux.go:349:
        starting container process caused "exec: \"argoexec\": executable file not
        found in $PATH": unknown'
      name: dag-diamond-1.A
      outputs:
        exitCode: "0"
      phase: Error
      progress: 1/1
      resourcesDuration:
        cpu: 0
        memory: 0
      startedAt: "2021-07-08T16:25:50Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod

For the request in this thread, the artifactRepository has populated s3 detail now.

[zijianjoy]

The existing executor-image is argoproj/argoexec:latest, which is different from workflow-controller, maybe I should use something different than docker.io/argoproj/workflow-controller:dev-6255 for --executor-image?

[terrytangyuan]

Yes it's a typo and should be argoexec image instead of workflow-controller.

[zijianjoy]

Thank you @terrytangyuan ! I validated that this dev image works for providing artifact info in status -> artifactRepositoryRef -> artifactRepository field. The new workflow run failed with 'Error (exit code 1): pods "dag-diamond-1-4281134405" is forbidden: User "system:serviceaccount:kubeflow:default" cannot patch resource "pods" in API group "" in the namespace "kubeflow"', but I think it is related to docker executor, and not related to this thread.

[Bobgy]

@zijianjoy the latest error message says kubeflow namespace's default service account do not have enough permissions.

The fix should be using a service account with enough permissions. E.g. pipeline-runner (deployed with kubeflow).

Change your workflow spec to add

spec:
  serviceAccountName: pipeline-runner

See https://argoproj.github.io/argo-workflows/workflow-rbac/

[zijianjoy]

@Bobgy Yes I have added service account from the initial run, which confused me why this error still shows up. You can search "serviceAccountName":"pipeline-runner" in this comment #6255 (comment).

[Bobgy]

@zijianjoy from your resource, looks like you added the field in metadata, but it should be in spec.

[zijianjoy]

@Bobgy Good catch! Yes I should add this field under spec, and the rerun is successful:

status:
  artifactRepositoryRef:
    artifactRepository:
      archiveLogs: true
      s3:
        accessKeySecret:
          key: accesskey
          name: mlpipeline-minio-artifact
        bucket: mlpipeline
        endpoint: minio-service.kubeflow:9000
        insecure: true
        keyFormat: artifacts/{{workflow.name}}/{{workflow.creationTimestamp.Y}}/{{workflow.creationTimestamp.m}}/{{workflow.creationTimestamp.d}}/{{pod.name}}
        secretKeySecret:
          key: secretkey
          name: mlpipeline-minio-artifact
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  - status: "True"
    type: Completed
  finishedAt: "2021-07-09T17:22:34Z"
  nodes:
    dag-diamond-1:
      children:
      - dag-diamond-1-4281134405
      displayName: dag-diamond-1
      finishedAt: "2021-07-09T17:22:34Z"
      id: dag-diamond-1
      name: dag-diamond-1
      outboundNodes:
      - dag-diamond-1-4197246310
      phase: Succeeded
      progress: 4/4
      resourcesDuration:
        cpu: 17
        memory: 3
      startedAt: "2021-07-09T17:21:41Z"
      templateName: diamond
      templateScope: local/dag-diamond-1
      type: DAG
    dag-diamond-1-4197246310:
      boundaryID: dag-diamond-1
      displayName: D
      finishedAt: "2021-07-09T17:22:26Z"
      id: dag-diamond-1-4197246310
      inputs:
        parameters:
        - name: message
          value: D
      name: dag-diamond-1.D
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/dag-diamond-1/2021/07/09/dag-diamond-1-4197246310/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 0
      startedAt: "2021-07-09T17:22:24Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod
    dag-diamond-1-4230801548:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4197246310
      displayName: B
      finishedAt: "2021-07-09T17:22:08Z"
      id: dag-diamond-1-4230801548
      inputs:
        parameters:
        - name: message
          value: B
      name: dag-diamond-1.B
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/dag-diamond-1/2021/07/09/dag-diamond-1-4230801548/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 2
        memory: 0
      startedAt: "2021-07-09T17:22:04Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod
    dag-diamond-1-4247579167:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4197246310
      displayName: C
      finishedAt: "2021-07-09T17:22:18Z"
      hostNodeName: gke-kfp-v2-compatible-pool-1-a19747a4-gflg
      id: dag-diamond-1-4247579167
      inputs:
        parameters:
        - name: message
          value: C
      name: dag-diamond-1.C
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/dag-diamond-1/2021/07/09/dag-diamond-1-4247579167/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 3
        memory: 0
      startedAt: "2021-07-09T17:22:04Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod
    dag-diamond-1-4281134405:
      boundaryID: dag-diamond-1
      children:
      - dag-diamond-1-4230801548
      - dag-diamond-1-4247579167
      displayName: A
      finishedAt: "2021-07-09T17:21:54Z"
      hostNodeName: gke-kfp-v2-compatible-pool-1-a19747a4-lr0h
      id: dag-diamond-1-4281134405
      inputs:
        parameters:
        - name: message
          value: A
      name: dag-diamond-1.A
      outputs:
        artifacts:
        - name: main-logs
          s3:
            key: artifacts/dag-diamond-1/2021/07/09/dag-diamond-1-4281134405/main.log
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 11
        memory: 3
      startedAt: "2021-07-09T17:21:41Z"
      templateName: echo
      templateScope: local/dag-diamond-1
      type: Pod
  phase: Succeeded
  progress: 4/4
  resourcesDuration:
    cpu: 17
    memory: 3
  startedAt: "2021-07-09T17:21:41Z"

[alexec]

@Bobgy good enough?

[Bobgy]

@alexec sure, that looks perfect!