Backup failed with error XAmzContentSHA256Mismatch

[leandreArturia]

What steps did you take and what happened:

Installed Velero 1.13.0 with CSI snapshot via Helm (vmware-tanzu/velero --version 6.0.0) . Here is my configuration :

configuration:
    backupStorageLocation:
    - name: default
      provider: aws
      bucket: xxx
      caCert: xxx
      config:
        s3Url: minio_underCA
        pulicUrl: minio_underCA
        region: minio
        s3ForcePathStyle: true
    volumeSnapshotLocation:
    - name: default
      provider: aws
      config:
        region: minio
        s3ForcePathStyle: true
    features: EnableCSI
  snapshotsEnabled: true
  credentials:
    existingSecret: velero-credential
  initContainers:
  - name: velero-plugin-for-aws
    image: velero/velero-plugin-for-aws:v1.9.1
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins
  - name: velero-plugin-for-csi
    image: velero/velero-plugin-for-csi:v0.7.0
    imagePullPolicy: IfNotPresent
    volumeMounts:
      - mountPath: /target
        name: plugins

I have a minIO deployed with self-signed certificate.

I was previously on velero 1.6 (with restic) without volumeSnapshot and the backup worked (with the same S3 credentials)

Now, when I try to do a backup, the volumeSnapshot work well but the backup fail:

time="2024-04-17T09:45:55Z" level=info msg="Summary for skipped PVs: []" backup=velero/jenkins-rd-backup-manual logSource="pkg/backup/backup.go:445"
time="2024-04-17T09:45:55Z" level=info msg="Backed up a total of 53 items" backup=velero/jenkins-rd-backup-manual logSource="pkg/backup/backup.go:449" progress=
time="2024-04-17T09:45:55Z" level=info msg="Setting up backup store to persist the backup" backup=velero/jenkins-rd-backup-manual logSource="pkg/controller/backup_controller.go:729"
time="2024-04-17T09:45:55Z" level=error msg="Error uploading log file" backup=jenkins-rd-backup-manual bucket=backup-k8s-dev error="rpc error: code = Unknown desc = error putting object backups/jenkins-rd-backup-manual/jenkins-rd-backup-manual-logs.gz: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17C708290FBD6CF3, HostID: 978e4e7e-71b7-4e48-84fa-5b1db1faf3d1, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed." error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:253" error.function="main.(*ObjectStore).PutObject" logSource="pkg/persistence/object_store.go:252" prefix=
time="2024-04-17T09:45:55Z" level=info msg="Initial backup processing complete, moving to FinalizingPartiallyFailed" backup=velero/jenkins-rd-backup-manual logSource="pkg/controller/backup_controller.go:743"
time="2024-04-17T09:45:55Z" level=error msg="backup failed" backuprequest=velero/jenkins-rd-backup-manual controller=backup error="rpc error: code = Unknown desc = error putting object backups/jenkins-rd-backup-manual/velero-backup.json: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 17C708290FE6506E, HostID: 978e4e7e-71b7-4e48-84fa-5b1db1faf3d1, api error XAmzContentSHA256Mismatch: The provided 'x-amz-content-sha256' header does not match what was computed." logSource="pkg/controller/backup_controller.go:288"
time="2024-04-17T09:45:55Z" level=info msg="Updating backup's final status" backuprequest=velero/jenkins-rd-backup-manual controller=backup logSource="pkg/controller/backup_controller.go:307"

What did you expect to happen:

To get a backup.

The following information will help us better understand what's going on:

bundle-2024-04-17-12-10-16.tar.gz

Environment:

• Velero version (use velero version): Server 1.13.0 Client 1.9.1
• Velero features (use velero client config get features): None
• Kubernetes version (use kubectl version): Server Version: v1.26.9+rke2r1
• Kubernetes installer & version: rke2r1
• Cloud provider or hardware configuration: bare metal
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[reasonerjt]

@leandreArturia May I know which version of Minio are you using?
I believe the change in aws-plugin v1.9.1 has been verified against Minio
cc @danfengliu

Additionally, please check aws-plugin v1.9.2 for the workaround to skip adding checksum header, you will need to make a change to the BSL.

[leandreArturia]

@leandreArturia May I know which version of Minio are you using? I believe the change in aws-plugin v1.9.1 has been verified against Minio cc @danfengliu

Additionally, please check aws-plugin v1.9.2 for the workaround to skip adding checksum header, you will need to make a change to the BSL.

I have a pretty old minio : minio version RELEASE.2022-08-25T07-17-05Z
I will check for the 1.9.2 aws plugin.

EDIT: It works with the aws plugin 1.9.2, thank you @reasonerjt

[reasonerjt]

@leandreArturia
Thank you for the update, let me keep this open until we double checked in our pipeline.

[danfengliu]

In Velero nightly pipeline, Minio version is 2024-04-18T19-09-19Z without CA, and pipeline passed, no such error occurred.

 containerStatuses:
  - containerID: containerd://edabecc8a9bede9288bfc293af7a408e115989ed1d6437098787df4cc96565a1
    image: docker.io/minio/minio:RELEASE.2024-04-18T19-09-19Z
    imageID: docker.io/minio/minio@sha256:036a068d7d6b69400da6bc07a480bee1e241ef3c341c41d988ed11f520f85124
    lastState: {}
    name: minio
    ready: true
    restartCount: 0
    started: true
    state:
      running:

[johnnyhuy]

Either options work - went with 2

1. Rolled back to 1.9.0 on the AWS plugin as mentioned the checksum verification wasn't added until the next patch
2. Updated the Backup Storage Location with checksumAlgorithm: "" to skip checks on Minio

   ---
   apiVersion: velero.io/v1
   kind: BackupStorageLocation
   metadata:
     name: example
   spec:
     config:
       region: minio
       s3ForcePathStyle: "true"
       s3Url: https://example.com
       # Disable checksumAlgorithm to avoid issues with Minio
       # https://github.com/vmware-tanzu/velero-plugin-for-aws/pull/197
       checksumAlgorithm: ""
     default: true
     objectStorage:
       bucket: example
     provider: aws
     credential:
       name: cloud-credentials
       key: cloud

I'm using a TrueNAS Minio service, on Velero 1.13.2.

Thanks for the fix!

[leandreArturia]

I think we can close this if you don't mind.
@johnnyhuy has summarized the 2 solutions.

[jaredkipe]

Same problem but with Linode Object storage, solution to empty the checksumAlgorithm works.

[mzimry]

@jaredkipe
having the same issue with linode, but with aws v1.9.0, so no checksumAlgorithm field and still getting errors. can you please share your BackupStorageLocation configs?

[Minipada]

Also having the same issue with Linode and disabled checksumAlgorithm

[fixed77]

maybe it's worth adding this parameter to official helm chart?