[velero-plugin-for-aws:v1.9.1] Brakes S3 compatible external storage location - NetApp ONTAP 9 Release 9.13.1P2

[leftyb]

Steps followed:
Upgrade to :
velero-plugin-for-aws:v1.9.1
Velero chart version : version: "6.7.0"
Velero version: appVersion: 1.13.2

Previous versions:
velero-plugin-for-aws:v1.8.2
Velero chart version : version: "5.2.2"
Velero version: appVersion: 1.12.3

What did you expect to happen:
Velero Backups properly stored to external S3 compatible storage location.
Before upgrade was working properly.

From velero logs:
When using checksumAlgorithm: "" Following error logs:
operation error S3: PutObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: The s3 command you requested is not implemented."

When NOT setting checksumAlgorithm so by default Velero BackupStorageLocation setting to CRC32
Getting following errors from velero logs (tested all available algorithms and got same error) :
operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

S3 Provider:
ONTAP 9
According to the ONTAP engineers AWS SDK V2 is supported

https://docs.netapp.com/us-en/ontap/s3-config/ontap-s3-supported-actions-reference.html#bucket-operations

similar to
#7543
#7828

[kaovilai]

So we need a way to set the signer to be unsigned.. and other values from https://github.com/aws/aws-sdk-go-v2/blob/84ca95e16adf482b8a80069d5ffa85814c7f61a2/aws/signer/internal/v4/const.go#L12

[kaovilai]

Can assign me and I'll double check where if any it can be set and publish some docs.

[kaovilai]

AWS SDK V2 is supported

If it is completely supported, all algorithms will be supported, including x-amz-content-sha256 which error mentions isn't supported here by ONTAP.

We will make best effort to allow configuration that will be compatible with others however.

[leftyb]

Hi,

Is there any update on the issue facing with velero-plugin-for-aws > v1.9.1. On pushing backups at ONTAP 9, which is S3 compatible?

Thank you.

[kaovilai]

This isn't currently on my short list of priorities so if you need something in less than 3 months I'd suggest trying prior to v1.9.1 AWS plugin

[leftyb]

Hi @kaovilai,
Thank you for the contact. I would like to ask, what is that you consider is the problem so that ONTAP 9 is not "allowing" Velero AWS plugin to connect and push backups? It seems that ONTAP 9 is supporting AWS SDK V2?
Thank you .

[kaovilai]

@leftyb ONTAP 9 is missing S3 compatible (at least not documented) Checking object integrity
checksum algorithms support.

As you discovered, velero with sdkv2 bump in plugin v1.9.1 uses CRC32 by default per aws documentation.

This is not available or documented at all in ONTAP docs.

So ONTAP 9 do not make available all s3 APIs. Velero with the SDKv2 will need to make accomodations to limit usage of features ONTAP 9 has not implemented.

It would help in the meantime for you to open a support case with ONTAP to get these implemented.

[leftyb]

Hi all,

I would like to update that issue is fixed using latest version NetApp Release 9.15.1P2.
We tested it and we can properly create backups and recover.
Issue can be considered as solved and can be closed.

Thanks for the support.