CSI snapshot not being taken (rook-ceph)

[RossWilliamson]

What steps did you take and what happened:
Trying to do a backup with CSI snapshot - Rook-ceph is in use. Have confirmed snapshots work with manual snapshot taken. When using velero the snapshot is skiped with the folllowing error.

time="2025-01-06T03:50:41Z" level=info msg="Skip action velero.io/csi-pvc-backupper for resource persistentvolumeclaims:tmp/rbd-pvc, because the CSI feature is not enabled. Feature setting is ." backup=velero/new5 logSource="pkg/backup/item_backupper.go:369" name=rbd-pvc namespace=tmp resource=persistentvolumeclaims
`
I'll put more details below. Deployment is via helm and I've tried both with and without features: EnableCSI - Both cases the same (actually when I don't have features: EnableCSI it still seems to claim it's enabled (See below)

No snapshot is taken

More logs around the error

time="2025-01-06T03:50:41Z" level=info msg="Executing takePVSnapshot" backup=velero/new5 logSource="pkg/backup/item_backupper.go:549" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="performing snapshot action for pv %!s(MISSING) as the snapshotVolumes is not set to false" backup=velero/new5 logSource="internal/volumehelper/volume_policy_helper.go:131"
time="2025-01-06T03:50:41Z" level=warning msg="VolumeSnapshotter plugin doesn't support data movement." backup=velero/new5 logSource="pkg/backup/item_backupper.go:584" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=velero/new5 logSource="pkg/backup/item_backupper.go:608" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=velero/new5 logSource="pkg/backup/item_backupper.go:612" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="zone info not available in nodeAffinity requirements" backup=velero/new5 logSource="pkg/backup/item_backupper.go:617" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="Unable to handle CSI driver: rook-ceph.rbd.csi.ceph.com" backup=velero/new5 cmd=/plugins/velero-plugin-for-aws logSource="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/volume_snapshotter.go:269" pluginName=velero-plugin-for-aws
time="2025-01-06T03:50:41Z" level=warning msg="No volume ID returned by volume snapshotter for persistent volume" backup=velero/new5 logSource="pkg/backup/item_backupper.go:641" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes volumeSnapshotLocation=default
time="2025-01-06T03:50:41Z" level=info msg="Persistent volume is not a supported volume type for Velero-native volumeSnapshotter snapshot, skipping." backup=velero/new5 logSource="pkg/backup/item_backupper.go:653" name=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 namespace= persistentVolume=pvc-17238d74-1f1b-45c5-a0c4-90df51cf1ea6 resource=persistentvolumes
time="2025-01-06T03:50:41Z" level=info msg="Executing custom action" backup=velero/new5 logSource="pkg/backup/item_backupper.go:357" name=rbd-pvc namespace=tmp resource=persistentvolumeclaims
time="2025-01-06T03:50:41Z" level=info msg="Skip action velero.io/csi-pvc-backupper for resource pe

What did you expect to happen:

Expected velero to create a snapshot and back it up to the minio s3 server

The following information will help us better understand what's going on:

bundle-2025-01-05-19-53-22.tar.gz

Environment:

• Velero version (use velero version):

Client:
Version: v1.15.1
Git commit: 32499fc
Server:
Version: v1.15.1

• Velero features (use velero client config get features):

features: EnableCSI

• Kubernetes version (use kubectl version):

Client Version: v1.31.4
Kustomize Version: v5.4.2
Server Version: v1.25.9+k3s1

• Kubernetes installer & version:

k3s

• Cloud provider or hardware configuration:
  self hosted minio

• OS (e.g. from /etc/os-release):

Ubuntu 22

HELM Values

configuration:
  uploaderType: kopia  # <----------------------------- Velero uses Kopia to upload data from volume snapshot
  backupStorageLocation:
    - bucket: velero  # <------------------- Name of bucket
      defaultVolumesToFsBackup: False
      provider: aws
      config:
        region: minio 
        insecureSkipTLSVerify: true
        s3ForcePathStyle: true  # <------------------ Only needed when S3-compatible storage is used instead of AWS S3 storage
        s3Url: https://192.168.1.22:9000 # <- Only needed when S3-compatible storage is used instead of AWS S3 storage
        checksumAlgorithm: ""
  volumeSnapshotLocation:
    - provider: aws
      config:
        region: minio
initContainers:
  - name: velero-plugin-for-aws   # <------------------ Velero plugin for AWS S3 bucket
    image: velero/velero-plugin-for-aws:v1.11.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins
deployNodeAgent: true
features: EnableCSI

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[RossWilliamson]

Ok a little more information - The CSI driver was not enabled even though I set it in values.yaml so I think this is an issue with the HELM not properly pushing the required command line arguments to the deployment:

                "containers": [
                    {
                        "args": [
                            "server",
                            "--uploader-type=kopia",
                            "--keep-latest-maintenance-jobs=3"
                        ],
                        "command": [
                            "/velero"
                        ],

Adding in the --features=EnableCSI to the args it now backs up as expected (still need to check restore)