fix(run): set pipeline run status as failed when component fails

[jvallesm]

Because

• When a component fails to execute due to, e.g., an invalid input, the status in the pipeline run is Completed.
• When a pipeline fails in the PreTriggerActivity process (load information to memory, resolve secrets and connections, initialise templates), the pipeline run doesn't contain the input or snapshot recipe.

This commit

• Marks the pipeline run and the component run as failed when any of the jobs in a component execution fails.
  • In the future we might want to have a more sophisticated way to handle errors. Perhaps a component can fail (e.g. another component is run given that condition as a fallback) and the pipeline execution can still be considered valid.
• Splits PreTriggerActivity in smaller activities and triggers the upload (recipe / input) activities as soon as the data is ready in memory.

[linear]

INS-6778 [Bug] Display "Completed" Status for Pipeline Runs with Intermediate Errors

[jvallesm]

QA ✅

Pipeline run status on component failure

Before (jq task on JSON component with invalid input JSON):

curl   http://localhost:8080/v1beta/namespaces/admin/pipelines/jq-error/runs\?pageSize\=100 \
--header "Content-Type: application/json" \
--header "Authorization: Bearer instill_sk_wHPobuR0xFB8nOiqGY4rXi2N2BqxCKQ1" |  jq '.pipelineRuns[0] | {status: .status, in: .inputs}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26542  100 26542    0     0   122k      0 --:--:-- --:--:-- --:--:--  122k
{
  "status": "RUN_STATUS_COMPLETED",
  "in": [
    {
      "json": "{\"foo: \"bar\"}"
    }
  ]
}

After:

curl   http://localhost:8080/v1beta/namespaces/admin/pipelines/jq-error/runs\?pageSize\=100 \
--header "Content-Type: application/json" \
--header "Authorization: Bearer instill_sk_wHPobuR0xFB8nOiqGY4rXi2N2BqxCKQ1" |  jq '.pipelineRuns[0] | {status: .status, in: .inputs}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 26542  100 26542    0     0   122k      0 --:--:-- --:--:-- --:--:--  122k
{
  "status": "RUN_STATUS_FAILED",
  "in": [
    {
      "json": "{foo: \"bar\"}"
    }
  ]
}

Missing input and recipe

Before (failed trigger due to wrong connection reference):

curl   http://localhost:8080/v1beta/namespaces/admin/pipelines/opp/runs\?pageSize\=100 \
--header "Content-Type: application/json" \
--header "Authorization: Bearer instill_sk_wHPobuR0xFB8nOiqGY4rXi2N2BqxCKQ1" |  jq '.pipelineRuns[0] | {at: .startTime, in: .inputs, recipe: .recipeSnapshot}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7337  100  7337    0     0  80033      0 --:--:-- --:--:-- --:--:-- 80626
{
  "at": "2024-11-14T08:56:05.247033Z",
  "in": [],
  "recipe": null
}

After:

curl   http://localhost:8080/v1beta/namespaces/admin/pipelines/opp/runs\?pageSize\=100 \
--header "Content-Type: application/json" \
--header "Authorization: Bearer instill_sk_wHPobuR0xFB8nOiqGY4rXi2N2BqxCKQ1" |  jq '.pipelineRuns[0] | {at: .startTime, in: .inputs, recipe: .recipeSnapshot}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12473  100 12473    0     0   185k      0 --:--:-- --:--:-- --:--:--  187k
{
  "at": "2024-11-14T10:14:53.695775Z",
  "in": [
    {
      "prompt": "How many inhabitants does Pamplona have?"
    }
  ],
  "recipe": {
    "component": {
      "openai-0": {
        "input": {
          "chat-history": null,
          "frequency-penalty": null,
          "images": null,
          "max-tokens": null,
          "model": "gpt-4o",
          "n": 1,
          "presence-penalty": null,
          "prompt": "${variable.prompt}",
          "response-format": {
            "type": "text"
          },
          "system-message": "You are a helpful assistant.",
          "temperature": 1,
          "top-p": 1
        },
        "setup": "${connection.oo}",
        "task": "TASK_TEXT_GENERATION",
        "type": "openai"
      }
    },
    "output": {
      "resp": {
        "title": "Response",
        "value": "${openai-0.output.texts}"
      }
    },
    "variable": {
      "prompt": {
        "format": "string"
      }
    },
    "version": "v1beta"
  }
}

[jvallesm]

Sorry, this wasn't meant to be the final state of the changes. I had (staged but) uncommitted changes to upload the recipe and input only under the TriggerFromAPI condition. They were meant to be squashed into this commit but I added them separately in the next commit to ease the review.