[bug] Pipeline metrics Invalid input error: Unknown execution spec

[dbg-raghulkrishna]

Environment

• How do you deploy Kubeflow Pipelines (KFP)?
  Kubeflow/manifest v1.6
  AKS v1.22

• KFP version: ml-pipeline/api-server:2.0.0-alpha.3

• KFP SDK version: '1.8.13'

Steps to reproduce

from kfp.components import InputPath, OutputPath, create_component_from_func
from kfp import dsl
def produce_metrics(
  # Note when the `create_component_from_func` method converts the function to a component, the function parameter "mlpipeline_metrics_path" becomes an output with name "mlpipeline_metrics" which is the correct name for metrics output.
  mlpipeline_metrics_path: OutputPath('Metrics'),
):
    import json

    accuracy = 0.1
    metrics = {
    'metrics': [{
      'name': 'accuracy-score', # The name of the metric. Visualized as the column name in the runs table.
      'numberValue':  accuracy, # The value of the metric. Must be a numeric value.
      'format': "PERCENTAGE",   # The optional format of the metric. Supported values are "RAW" (displayed in raw format) and "PERCENTAGE" (displayed in percentage format).
    }]
    }
    with open(mlpipeline_metrics_path, 'w') as f:
        json.dump(metrics, f)

produce_metrics_op = create_component_from_func(
    produce_metrics,
    base_image='python:3.7',
    packages_to_install=[],
    output_component_file='component.yaml',
)
from kfp import dsl
import os, kfp
@dsl.pipeline(
    name='KF iris PoC',
    description='Simple TF'
)
def test():
     task=produce_metrics_op()
import os,kfp
with open(os.environ['KF_PIPELINES_SA_TOKEN_PATH'], "r") as f:
    TOKEN = f.read()
client = kfp.Client(host='http://ml-pipeline.kubeflow.svc.cluster.local:8888',existing_token=TOKEN) 

client.create_run_from_pipeline_func(
    test,
    mode=kfp.dsl.PipelineExecutionMode.V1_LEGACY,
    arguments={}
   
)

Expected result

Materials and reference

https://www.kubeflow.org/docs/components/pipelines/sdk/pipelines-metrics/

I cannot see the pipeline metrics in the UI but the pipeline artifact is logged i am getting this error in ml-pipeline

I0912 09:12:02.846997       8 error.go:259] Invalid input error: Unknown execution spec
InternalServerError: failed to unmarshal workflow

Labels

---

Impacted by this bug? Give it a 👍.

[MatthiasCarnein]

I'm experiencing the same issue. It seems that the issue was introduced in the alpha.3 release. Metrics are successfully picked up when downgrading the ml-pipeline deployment to the previous image gcr.io/ml-pipeline/api-server:2.0.0-alpha.2

[juliusvonkohout]

@TobiasGoerke and myself can reproduce this too. @kimwnasptd @annajung this is another serious regresseion and candidate for Kubeflow 1.6.1
@Linchin @james-jwu Is this fixed in a newer release that we can use for 1.6.1?

[chensun]

I'll take a look.

Slightly off the topic, per #8256 (comment), this is a regression in KFP 2.0.0a3. While last time we communicated, KFP 2.0.0a2 should be the version to go with Kubeflow 1.6 release (kubeflow/manifests#2198 (comment)).

Given the natural of alpha phase release, I think we probably shouldn't automatically chase the latest alpha release available, they may not be very well tested. So while I investigate this issue, can we pin back to KFP 2.0.0a2 in KF 1.6? I assume that version is possibly tested more thoroughly with KF 1.6 RC releases. WDTY? @annajung @zijianjoy @gkcalat

[chensun]

can we pin back to KFP 2.0.0a2 in KF 1.6? I assume that version is possibly tested more thoroughly with KF 1.6 RC releases.

I take this back, checked the change history and it appears we never included KFP 2.0.0a2 in any KF 1.6 RC release.
Let me focus on this specific issue first.

[chensun]

can we pin back to KFP 2.0.0a2 in KF 1.6? I assume that version is possibly tested more thoroughly with KF 1.6 RC releases.

I take this back, checked the change history and it appears we never included KFP 2.0.0a2 in any KF 1.6 RC release. Let me focus on this specific issue first.

Discussed with @gkcalat offline. We'll release with 2.0.0a5 with the fix, and tentatively use this version for KF 1.6 patch release.
That being said, I still think in future KF release, we shouldn't automatically pick the latest KFP alpha release available.

[annajung]

Thanks @chensun, do you know when we can expect 2.0.0.a5?

[chensun]

Thanks @chensun, do you know when we can expect 2.0.0.a5?

@annajung 2.0.0a5 release is done: https://github.com/kubeflow/pipelines/releases/tag/2.0.0-alpha.5