can the CSI side car image be hosted in a private repository and referenced, for security purpose.

[ashish01987]

Some organizations may not allow to pull images from public reporsitory directly into the cluster. May be they prefer to host it privately and do regular security scans.
is it possible to host gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter:v0.1.4-gke.1@sha256:442969f1e565ba63ff22837ce7a530b6cbdb26330140b7f9e1dc23f53f1df335 privately and side car reference to private repository ?

[ashish01987]

Any updates on this ?

[songjiaxun]

For now we do not support private sidecar container images. The CSI driver will check if the sidecar container image uses the registry gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter. The GKE will do all the security scans, and GKE will update the sidecar container images regularly to resolve the security vulnerabilities.

[Siegfriedk]

@songjiaxun we actually run in the same issue and even considered patching the driver itself.

None of our images are pulled from remote and all images GKE manages don't create an issue in our side as no other image than this is running in customer namespace

[songjiaxun]

Hi @Siegfriedk and @ashish01987 , can I learn more about your use cases? It seems like you are OK with the GKE managed images in the kube-system namespace, but you want all the images in customer namespace from a private container registry, is my understanding correct?

[Siegfriedk]

@songjiaxun yes. Customer namespaces have gatekeeper policies in place to make sure no external images are getting pulled. And we also make sure that system critical things are coming from our local registry (gcr) after scanning.

[ashish01987]

Any plan to add support for this ?

[songjiaxun]

Hi @ashish01987 , @Siegfriedk the support to allow the webhook to inject a sidecar consuming a customized container image has a lower priority, unfortunately.

I'd like to learn more about the workflow here, so I am proposing a workflow here as a workaround. And if this works for your use case, I can make the change real quick.

1. GKE publishes the official sidecar image, e.g. gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter:v0.1.4-gke.1@sha256:xyz.
2. You pull the image and re-publish the image to your private container registry, e.g. my.private.registry/gcs-fuse-csi-driver-sidecar-mounter:v0.1.4-gke.1@sha256:xyz
3. In your workload, you manually inject the sidecar container using the private container image. GKE will provide guidance for you to do this.
4. I change the webhook code to bypass the container image validation, so that the manually injected sidecar will work.

Please let me know your thoughts, thank you!

[Siegfriedk]

@songjiaxun that would be great, we already thought about patching it ourselves

[ashish01987]

"In your workload, you manually inject the sidecar container using the private container image. GKE will provide guidance for you to do this."
Will this require elevating any privilege's in any of my pods / containers ?
May be you can briefly share the steps for this, let me check if that should be possible/allowed in customer deployments.

[songjiaxun]

I am trying to propose to add a new pod annotation to allow you specify a private sidecar container image. But I have to go through some internal security review. Will keep you posted. Thanks!

[songjiaxun]

Hi @Siegfriedk , @ashish01987

I got a response from our GKE security team -- users will need to manually inject the sidecar spec using the private images.

Please refer to this comment as an example. When we support this pattern, we will update the official documentation to support manual sidecar injection. Thank you!

[songjiaxun]

We finalized the workflow as the following. @Siegfriedk , @ashish01987 please let me know if this looks good to you

1. GKE provides documentation to let the users figure out which GCS FUSE CSI driver version the cluster is using, and provides a GKE-built sidecar container image.
2. The user pulls the GKE-built sidecar container image, and pushes it to a private container registry.
3. The user manually specifies the sidecar container spec. For example:

apiVersion: v1
kind: Pod
metadata:
 name: my_pod
 namespace: my_namespace
 annotations:
   gke-gcsfuse/volumes: "true" # required
spec:
 containers:
 - name: gke-gcsfuse-sidecar # <- manually injected
   image: private-registry/gcs-fuse-csi-driver-sidecar-mounter:my-tag
 - image: nginx
   name: nginx
   volumeMounts:
   - name: gcp-cloud-storage-csi-ephemeral
     mountPath: /data
 serviceAccountName: my_k8s_sa
 volumes:
 - name: gcp-cloud-storage-csi-ephemeral
   csi:
     driver: gcsfuse.csi.storage.gke.io
     volumeAttributes:
       bucketName: my-bucket-name

4. The GKE webhook takes the container image from the Pod spec and injects the rest of the sidecar container.

[ashish01987]

Thanks @songjiaxun. Looks good to me.
Just one question,
Is it still necessary to have "name: gke-gcsfuse-sidecar" at "0" th index in container list ?
since now the container is not injected at run time, but through yaml definitions.
The customer might have some patches which they generally apply for some customization. This might be issue, if some of the patch is assuming some other container at 0th position.
Can we change the order of gke-gcsfuse-sidecar may append to list of containers ?

[songjiaxun]

To be clear, our proposal will "partially" inject the sidecar container. The webhook only reads the sidecar container from your sidecar container spec and ignores other customization, and prepare for the rest of the sidecar container. The sidecar container has to be at the "0" index in container list. Even though you specified the sidecar container at non-0 index, as long as you have the pod annotation gke-gcsfuse/volumes: "true", our webhook will do its job to put it back at the "0" index.

In order to completely manually inject the sidecar container, you can remove the Pod annotation gke-gcsfuse/volumes: "true", and manually inject the sidecar container wherever you like, but this is not officially supported/recommended by GKE. See #20 for details.

Also, FYI, we are working on adopting the native k8s sidecar container feature, targeting GKE 1.29. By then, the sidecar container will be injected as an init container.

[ashish01987]

Thanks This answers my question. As long as the injection pattern remains same as earlier, i dont see this as any breaking change.

[ashish01987]

In our deployment we might have several pod deployments who want to push their file system data to GCS through the csi driver, for DR purpose.
gke-gcsfuse-sidecar definition will be go as a patch to these deployments who want to put files on GCS. For customer who want to opt in this feature.
The patch will append this gke-gcsfuse-sidecar to existing list of container in the target deployment along with patching gke-gcsfuse/volumes: "true"
something like
patch: |-

• op: add
  path: /../../../containers/-
  value:
  name: gke-gcsfuse-sidecar # <- manually injected
  image: private-registry/gcs-fuse-csi-driver-sidecar-mounter:my-tag

And the your webhooks should bring the sidecar container to 0th position at runtime.

I guess this approach should work.

[ashish01987]

Any tentative timeline when official documentation is available for this approach ?

[songjiaxun]

@ashish01987 , the release is still in progress. When the release is done, I will update the official documentation. ETA is 2024-01-25.

[songjiaxun]

The roll out was completed. Updated the doc: https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#private-image

And please refer to the requirements to upgrade your cluster to the required versions:
https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/cloud-storage-fuse-csi-driver#requirements