[feature] Support PV/PVCs as an alternative to Object Storage for KFP

[HumairAK]

Feature Area

What feature would you like to see?

Currently there is a hard dependency in the launcher for object storage used for artifact passing. Users should have the option to configure a PV instead. This would drastically increase the different types of backing storage KFP can support by virtue of taking advantage of K8S storage class abstraction.

Note: Object storage is also used by the API Server for storing the Pipeline IR, but this dependency should be removed via: #10509

What is the use case or pain point?

Many end users operating in airgapped (or other) environments that do not have an enterprise ready object store solution, find it an unnecessary requirement when they have other means of storage hooked in to their k8s platform they would rather utilize instead.

Is there a workaround currently?

No

---

Love this idea? Give it a 👍.

[Tomcli]

One thing to consider in this case is the access control list. With pure PVC, you have no API to control who can access the artifacts. Since pipelines artifacts can be cached, not all the artifacts will be in the same pipeline subpath which can create extra complexity.

I would recommend to support another client that can support the some storage API where it can pass the artifact files into PVC without mounting to the task pod.

[gregsheremeta]

I would recommend to support another client that can support the some storage API where it can pass the artifact files into PVC without mounting to the task pod.

Hi @Tomcli, can you rephrase or elaborate? I didn't understand this part.

[Tomcli]

I would recommend to support another client that can support the some storage API where it can pass the artifact files into PVC without mounting to the task pod.

Hi @Tomcli, can you rephrase or elaborate? I didn't understand this part.

By default file storage and pvc have no API. Therefore, projects like minio has the S3 API to help take the artifact file in the form of HTTP etc and store into the mounted pvc along with ACL to manage who owns the file. Without such API, if we volume mount every task pod to the PVC, we will be creating active volume connections per each task.

[Tomcli]

We have discussed in the past for the Tekton community on how it should implemented.
tektoncd/pipeline#4012

[juliusvonkohout]

Actually KFP v1 still supports a PVC as object passing method natively. They just removed it in V2.

[chensun]

Actually KFP v1 still supports a PVC as object passing method natively. They just removed it in V2.

Using PVC for data passing is supported in v2: https://www.kubeflow.org/docs/components/pipelines/v2/platform-specific-features/

[juliusvonkohout]

Actually KFP v1 still supports a PVC as object passing method natively. They just removed it in V2.

Using PVC for data passing is supported in v2: https://www.kubeflow.org/docs/components/pipelines/v2/platform-specific-features/

This is manual data passing, not comparable to the old automatic data_passing_method
https://kubeflow-pipelines.readthedocs.io/en/1.8.22/source/kfp.dsl.html#kfp.dsl.PipelineConf.data_passing_method it is not even documented in V1 but it is in the code and works

@property
    def data_passing_method(self):
        return self._data_passing_method

    @data_passing_method.setter
    def data_passing_method(self, value):
        """Sets the object representing the method used for intermediate data
        passing.

        Example:
          ::

            from kfp.dsl import PipelineConf, data_passing_methods
            from kubernetes.client.models import V1Volume, V1PersistentVolumeClaimVolumeSource
            pipeline_conf = PipelineConf()
            pipeline_conf.data_passing_method =
            data_passing_methods.KubernetesVolume(
                volume=V1Volume(
                    name='data',
                    persistent_volume_claim=V1PersistentVolumeClaimVolumeSource('data-volume'),
                ),
                path_prefix='artifact_data/',
            )
        """
        self._data_passing_method = value

[HumairAK]

as @juliusvonkohout mentioned above, while users could do this for their individual pipelines

this issue requests a feature that does this automatically by default at the kfp level so users don't have to worry about this abstraction

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/frozen
/lifecycle-frozen

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/lifecycle frozen

this is still relevant and a regression from V1

[HumairAK]

this is still relevant and a regression from V1

I might be missing something, how is this a regression from V1?

[juliusvonkohout]

this is still relevant and a regression from V1

I might be missing something, how is this a regression from V1?

Because it was supported in V1 and removed in V2 as mentioned above.

[HumairAK]

Once again this issue is about a blanket support for a kfp install that utilizes PVC over object store. Though what you seem to be talking about does not sound entirely unrelated, however it seems like it will require sdk changes as well. The use cases while related are slightly different:

1. As a user I want to specify within my pipeline code a PVC to utilize for data passing
2. As a user I want to deploy KFP so that a pvc (specified at deploy time) is utilized for data passing instead of object store by default for all pipeline runs

I suppose the assignee is welcome to address both of these, however afaik only (1) is a regression, (2) is a new feature.