Support Multi-Region Access Points for S3 sources

[zeleena]

Feature Request

Environment

What hardware/cloud provider/hypervisor is being used to run Ignition? AWS

Desired Feature

We can only provide one source for the config, which can lead to single points of failure especially when relying on S3 buckets tied to one region. Luckily, AWS already implements S3 bucket region fallback with Multi-Region Access Points. Ignition should support multi-region access points as an S3 source.

Multi-region access points use the form arn:aws:s3::<account-id>:accesspoint/<alias>.mrap according to https://docs.aws.amazon.com/AmazonS3/latest/userguide/MultiRegionAccessPointRequests.html

The current S3 source would need to be modified to allow for S3 access point URLs in ARN format (opaque URLs starting with s3:arn), and #1264 would accomplish this.

Next, AWS sdk support for multi-region access points is only in aws-sdk-go-v2, so aws-sdk-go (v1) would have to be converted to aws-sdk-go-v2.

Lastly, modifying the code to support downloading sources from multi-region access points.

[cgwalters]

This looks strongly related to #1100 and specifically #1104 right?

[zeleena]

#1100 is different because it asks for support for gzip'd user data, while this issue is pulling the config from S3 therefore not limited by the 16384 byte user data limit.

#1104 would accomplish this in a different way, but the feature I propose here wouldn't need to modify the config specification. Ignition wouldn't have to handle the fallback mechanism. Additionally, this issue hasn't had any activity for a year now.

For what it's worth, Lyft already has a solution and has been running support for multi-region access points in production for about a month now.

[bgilbert]

Both of those bugs are related. #1100 was an attempt to avoid hosting configs > 16 KiB in S3, in order to avoid the regional single point of failure. #1104 was proposed as an alternative, but its implementation in #1176 has stalled. It turned out to be somewhat intrusive, and doesn't mesh well with config-merging semantics.

This proposal sounds like a good alternative, for S3 at least. Does the deployed Lyft solution use a modified version of Ignition as proposed here?

[zeleena]

Sure, they're the same underlying issue for our specific use case but correspond to different changes in ignition.

Yes, the deployed Lyft solution uses a modified version of Ignition proposed here. However we'd really like to sync with upstream since we are running a development version of ignition, and the first part of getting upstream to support this is the S3 ARN support in #1264 which needs another review. The next part would be using aws-sdk-go-v2 which has multi-region access point support, and we have that already implemented as well.

[bgilbert]

If this eventually gets implemented, kola test fixtures are already provisioned and there's a draft PR to add them to the ext.config.ignition.resource.authenticated-s3 kola test:

• WIP: tests/kola/ignition: test S3 multi-region access points fedora-coreos-config#1745

I'll close that PR for now, but it's available for future reference when needed.