[Issue #3097] Adds configuration for S3 CDN

[coilysiren]

Summary

Probably finishes #3097

Time to review: 5 mins

Changes proposed

Adds configuration to the CDN that allows it to operate in 4 modes:

1. With no fronting domain name, and an ALB origin (eg. frontend dev)
2. With a fronting domain name, and an ALB origin (eg. frontend prod)
3. With no fronting domain name, and a S3 origin (eg. API dev)
4. With a fronting domain name, and a S3 origin (eg. API prod)

Context for reviewers

It's complicated in here 😅

Testing

I tested by deploying to API dev and frontend dev

[chouinar]

Doesn't need to be in this PR, but we'll want this value to be an env var the API gets.

The plan I had for this was just to swap out the s3 bucket for the CDN domain name on the fly in the API (eg. s3://public-bucket-1234/path/to/file.txt becomes https://the-cdn-name/path/to/file.txt)

[coilysiren]

I can build that into this PR!

[mdragon]

@coilysiren, just to make sure I'm clear.

1. The references to API in the PR itself (not the code) are about CDN->S3 because our "API" is the service that owns the S3 buckets for files that we're looking to front with Cloudfront? So we have CDN to ALB to CDN to S3 support following this change.
2. In the scenario where frontend turns on ALB CDN, and API turns on S3 CDN, do we have 2 CDNs? Or is this able to apply both sets of configuration into a single CDN instance?

[coilysiren]

In the scenario where frontend turns on ALB CDN, and API turns on S3 CDN, do we have 2 CDNs? Or is this able to apply both sets of configuration into a single CDN instance?

2 CDNs, which is my preference.

[coilysiren]

The references to API in the PR itself (not the code) are about CDN->S3 because our "API" is the service that owns the S3 buckets for files that we're looking to front with Cloudfront? So we have frontend CDN to ALB and API CDN to S3 to support following this change.

Comment edited for clarity, and yes.

[mdragon]

2 CDNs, which is my preference.

Ok, great, that's what I was hoping, but just couldn't put all the terraform together in my head to be sure. I think that's going to be very beneficial as we look to do mass-invalidation on deploy (which wouldn't need to touch the S3 CDN at all for example).