Can't access artifacts via UI when artifacts come from template referenced by templateref

[Transmitt0r]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I expect to be able to download/view artifacts via the server-UI. This is only possible with artifacts which originate from templates that are defined within the same workflow/workflowtemplate.

Version

3.4.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Working Example:

# This example demonstrates the ability to pass artifacts
# from one step to the next.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  serviceAccountName: argo-workflow
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["sleep 1; cowsay hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      - name: hello-art
        path: /tmp/hello_world.txt

  - name: print-message
    inputs:
      artifacts:
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]

Failing example:

# This example demonstrates the ability to pass artifacts
# from one step to the next.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  serviceAccountName: argo-workflow
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        templateRef:
          name: whalesay-template
          template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: print-message
    inputs:
      artifacts:
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]

---
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: whalesay-template
spec:
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["sleep 1; cowsay hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      - name: hello-art
        path: /tmp/hello_world.txt

Logs from the server

time="2022-09-20T12:32:16.527Z" level=info msg="Get artifact file" artifactName=output namespace=argo nodeId=test-build-n5rw5-3520357635 workflowName=test-build-n5rw5
time="2022-09-20T12:32:16.542Z" level=error msg="Artifact Server returned internal error" error="no template found by the name of '' (which is the template associated with nodeId 'test-build-n5rw5-3520357635'??"
time="2022-09-20T12:32:16.542Z" level=info duration=15.435175ms method=GET path=/artifact-files/argo/workflows/test-build-n5rw5/test-n5rw5-3520357635/outputs/output size=22 status=500
time="2022-09-20T12:32:16.590Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2022-09-20T12:32:16Z" grpc.time_ms=9.277 span.kind=server system=grpc
time="2022-09-20T12:32:16.591Z" level=info duration=10.740922ms method=GET path=/api/v1/workflows/argo/demo-ascet-vecu-build-n5rw5 size=19328 status=0
time="2022-09-20T12:32:17.576Z" level=info duration=3m53.079937295s method=GET path=/api/v1/workflow-events/argo size=19516 status=0

Reference: Discussion about this in Slack
Examples.zip

[brianloss]

I looked into this a little bit, and it seems this happens if the artifact repository is specified in the workflow controller configmap and archiveLogs is not set. If archiveLogs is true, then it works. Or, if the default artifact repository is set via the artifact-repositories configmap (and the workflows.argoproj.io/default-artifact-repository annotation), it works whether or not archiveLogs is set.

[Transmitt0r]

@brianloss so I'm not using the defaultArtifactRepository in the controller configmap, but via the artifact-repositories configmap with the default annotation.

[brianloss]

I looked into this a little more, and it seems there is an issue where there is non-determinism as to what shows up in the artifacts panel when running the artifact passing example workflow. When you inspect the workflow k8s resource and look at the nodes, sometimes the generate-artifact node appears before the consume-artifact node and sometimes it is the other way around. This corresponds to the behavior in the UI where when clicking on the "hello-art.tgz" that shows between the generate-artifact and consume-artifact nodes on the graph, sometimes the artifact panel shows the hello-art output of generate-artifact and sometimes it shows the message input to consume-artifact. When the former happens and it shows hello-art and is attempting to retrieve the output artifact, that's when the internal server error happens. So that's why it was working sometimes and not others with the artifact passing example. I find that non-determinism in the UI confusing, but it should probably be a separate issue.

There is still the underlying bug, and it appears to be in the artifact server when retrieving artifacts for a workflow node that does not have the templateName field set but instead has the templateRef field set. The artifact server is not handling that case.

[juliev0]

@brianloss Is it okay if we assign you this one and this one? (which is likely the same)

[juliev0]

Here's what I ran locally which created a Node of type="Pod" that had no TemplateName or TemplateRef when I looked at the status.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: dag-inline
  annotations:
    workflows.argoproj.io/description: |
      This examples demonstrates running a DAG with inline templates.
    workflows.argoproj.io/version: ">= 3.2.0"
spec:
  entrypoint: main
  templates:
    - name: main
      dag:
        tasks:
          - name: a
            inline:
              container:
                image: argoproj/argosay:v2

kind: Workflow
metadata:
  annotations:
    workflows.argoproj.io/pod-name-format: v1
  creationTimestamp: "2022-09-20T07:15:38Z"
  generateName: workflow-push-weckdengeparden-
  generation: 19
  labels:
  namespace: argo
spec:
  arguments:
    parameters:
    - name: repository
      value: [email protected]:gepaplexx-demos/weckdengeparden.git
    - name: revision
      value: refs/heads/main
  entrypoint: main
  workflowTemplateRef:
    clusterScope: true
    name: dag-inline

I'm not sure what to make of that.

[brianloss]

@juliev0, your example doesn't actually have any artifacts, but I tried combining the examples above to add artifacts and the artifact server seems to do the right thing, though I'm not exactly sure how. The template with an empty name is matching in the storedTemplates section of the status. However, when I click on the tgz node in the graph of the UI, that does give me an error:

TypeError: Cannot read properties of undefined (reading 'archiveLocation')
    at http://localhost:8080/main.9400eea421eeb44eecfc.js:385292:41
    at commitHookEffectListMount (http://localhost:8080/main.9400eea421eeb44eecfc.js:294419:26)
    at commitPassiveHookEffects (http://localhost:8080/main.9400eea421eeb44eecfc.js:294457:11)
    at HTMLUnknownElement.callCallback (http://localhost:8080/main.9400eea421eeb44eecfc.js:274876:14)
    at Object.invokeGuardedCallbackDev (http://localhost:8080/main.9400eea421eeb44eecfc.js:274925:16)
    at invokeGuardedCallback (http://localhost:8080/main.9400eea421eeb44eecfc.js:274980:31)
    at flushPassiveEffectsImpl (http://localhost:8080/main.9400eea421eeb44eecfc.js:297541:9)
    at unstable_runWithPriority (http://localhost:8080/main.9400eea421eeb44eecfc.js:340644:12)
    at runWithPriority$1 (http://localhost:8080/main.9400eea421eeb44eecfc.js:285727:10)
    at flushPassiveEffects (http://localhost:8080/main.9400eea421eeb44eecfc.js:297508:12)

If I click on the task node and download from the list of artifacts there, that's where it does the right thing. So aside from the UI issue, I'm not sure if there's still a problem to fix or not. In either case, we might have exceeded my knowledge of the code. :)

[juliev0]

The template with an empty name is matching in the storedTemplates section of the status.

I noticed that as well! Pretty weird. I'd say if you just want to submit a PR for the issue from the original bug (where templateRef is there), I think that's good. Thanks!

[brianloss]

@juliev0 #9648 should have that. I checked that I can run the examples in both this issue and #9644

[juliev0]

#9648

awesome! I will try to review your PR when I have time