aws-certificatemanager: Support construct to create Certificate from any region, similar to EdgeFunction

[Xarno]

Describe the feature

As requested and liked many times please have the Certificate Construct accept a region setting again, as it has had in DnsValidatedCertificate.
#9274 (comment)

Use Case

Create Cloudfront distributions in one region, create the corresponding certificate in us-east-1.
Create Cognito Userpools with a custom certificate as mostafafarzaneh mentioned in the linked issue.

Proposed Solution

No response

Other Information

Also the current documentation is wrong.
https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.Certificate.html#validation mentions a „region“ property which does not exist in https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.CertificateValidation.html or https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager.CertificationValidationProps.html

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.76.0

Environment details (OS name and version, etc.)

Mac OS 13.3

[peterwoodworth]

This is how we recommend doing it now https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager-readme.html#cross-region-certificates. If this doesn't work then let us know

[alejandro-gstack]

@peterwoodworth, The way that AWS is proposing managing the certificate in a different stack is not providing a good developer experience because instead of defining in the same CDK stack, we will need to create two stacks, one for creating the ACM certificate in the us-east-1 region that requests for example Cloudfront and then reference it in the other stack built in the desired region.

[peterwoodworth]

I'm curious what the negative experience is exactly with needing to declare another stack in your app, is there a use case where this isn't trivial?

[alejandro-gstack]

Hi @peterwoodworth, consider that you are creating templates (constructs in CDK) to reduce the complexity of deploying infrastructure like a static website (s3/cloudfront/route53/certificate). With the previous approach, developers will create a CDK stack and declare the template and the resources needed in one stack.

Now, they will need to manage two stacks and one stack that only will contain a certificate, and they will need to deploy them in a specific order.

Do you know what would be the suitable infra definition to encapsulate the creation of the certificate in (us-east-1) and all the other resources in a different region? Can I attach the two CDK stacks and programmatically define the order in which they will be deployed?

Checking the documentation, we will need to deploy the CDK stacks separately and in a particular order.

import { aws_cloudfront as cloudfront, aws_cloudfront_origins as origins } from 'aws-cdk-lib';
declare const app: App;

const stack1 = new Stack(app, 'Stack1', {
  env: {
    region: 'us-east-1',
  },
  crossRegionReferences: true,
});
const cert = new acm.Certificate(stack1, 'Cert', {
  domainName: '*.example.com',
  validation: acm.CertificateValidation.fromDns(PublicHostedZone.fromHostedZoneId(stack1, 'Zone', 'ZONE_ID')),
});

const stack2 = new Stack(app, 'Stack2', {
  env: {
    region: 'us-east-2',
  },
  crossRegionReferences: true,
});

new cloudfront.Distribution(stack2, 'Distribution', {
  defaultBehavior: {
    origin: new origins.HttpOrigin('example.com'),
  },
  domainNames: ['dev.example.com'],
  certificate: cert,
});

[alejandro-gstack]

This is a great example of the use case I'm mentioning; what will be the right way to simplify this construct definition using multi-stack deployments?

By the way, I'm not familiar with multi-stack deployments.

https://github.com/aws-samples/aws-cdk-examples/blob/master/typescript/static-site/static-site.ts#L24

[peterwoodworth]

All you should need to do is create a separate stack for your Certificate that deploys into us-east-1, I don't think this would add much complexity in any use case I can imagine - However you define this is up to you. If you're unfamiliar with creating apps with multiple stacks, please see our docs on creating an app with multiple stacks, and having stacks depend on another stack

[peterwoodworth]

For those coming to this thread, please see this comment left by Cory explaining why DnsValidatedCertificate was deprecated. We still stand by this decision. However, as Cory notes in this comment, we would be on board with supporting a construct that handles the creation of the stack in us-east-1 for you. Relabelling to p1, as this is clearly a desired feature

[blimmer]

@peterwoodworth , I think the idea of a construct that assists with the "create a stack in a different region" workaround is a good compromise! That will make handling the deprecation warning much simpler.

[ckifer]

Another use-case for this where creating another stack isn't trivial is when managing the HostedZone in the same stack Cloudfront is in. I can't just create a us-east-1 stack with a Certificate because my HostedZone doesn't exist until it is created in the other-region stack - and I need its information for the cert validation.

I would need to create 3 stacks dependent on one another: HZ creation (in say us-west-2) -> Cert creation (in us-east-1) -> Cloudfront distro (back to us-west-2). Unless I'm missing something and there is an easier way to do that.

edit: I guess moving the HZ setup to the us-east-1 stack solves this. But still

[trautonen]

I've created a construct that handles this with a custom resource: https://github.com/trautonen/cdk-dns-validated-certificate
The construct handles the two most common cases:

• cross account hosted zone based dns validation
• cross region certificates (for example for CloudFront)

It's still lacking alias support, but I'm open to contributions and might have time to add it soon.

[guentherwieser]

Another use case that previously worked with DnsValidatedCertificate, and no longer works when you use multi-stack deployments, is this one here: #26714

We build the certificate's hostname from values from our Parameter Store in eu-central-1, but this fails with multi-stack-deployments with the error "Cross stack references are only supported for stacks deployed to the same account or between nested stacks and their parent stack".

[fdx-sachit]

CDK: "aws-cdk": "2.95.0",
Language: Typescript

I have a static site deployed in Sydney region. I have got a s3 bucket and cloudfront as a pattern. I am also trying to assign it a ssl certificate using ACM. I have a hosted zone in the global region. Now given my app is in Sydney region, and ACM only in us-east-1, i am trying to use the suggested way of doing this.

( I followed this tutorial https://www.subaud.io/blog/deploy-a-cross-region-spa-with-cloudfront )

I am trying the above suggestion on multi region stack. So here is what i've done:

• Initialise a cdk app

const app = new App({
    context: {
        [PERMISSIONS_BOUNDARY_CONTEXT_KEY]: {
            name: PERMISSION_BOUNDARY_NAME,
        },
    },
});

Please note the permission boundary that will become the problem i am having.

• Then i initialise a stack in us-east-1 for the certificate

const certificateStack = new CertificateStack(app, "CertificateStack", {
    env: {
        account: accountId,
        region: 'us-east-1',
    },
    synthesizer: new DefaultStackSynthesizer({
        qualifier: "...",
    }),
    appDomain: ...,
    primaryDomain: ...,
    crossRegionReferences: true,
});

• I have a second stack for my static site deployment as followes

const staticSite = new InfraCoreStack(app, "InfraCoreStack", {
    env: {
        account: accountId,
        region: app.node.tryGetContext("accountId") ?? "ap-southeast-2",
    },
    synthesizer: new DefaultStackSynthesizer({
        qualifier: "....",
    }),
    certificate: certificateStack.siteCertificate,
    crossRegionReferences: true,
});

Now when i synthesise this, it does that without error. But the problem is, because cdk is bootstrapped using a permission boundary, the certificate stack is not attaching that permission boundary to the IAM Role within the certificate stack. And when i try to deploy it, it fails to create the stack with permission errors.

API: iam:CreateRole User: arn:aws:sts::xxxx:assumed-role/xx-xxx-xxxx-xxxxx-us-east-1/AWSCloudFormation is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::xxxx:role/CertificateStack-CustomCrossRegionExportWriterCust-xxxx because no permissions boundary allows the iam:CreateRole action

Has anyone encountered this ? If so, can someone point me in the direction I need to fix this ? It will be a shame if i can't do it this way. Defeats the purpose of me trying to automate this process when I might have to have a certificate manually created and referenced as a parameter.

Thanks

[blimmer]

@peterwoodworth , not sure how much collaboration CDK has with the CFN team within Amazon, but an alternative to working around this in CDK would be to push forward on this CFN request: aws-cloudformation/cloudformation-coverage-roadmap#523

[pergardebrink]

I'm curious what the negative experience is exactly with needing to declare another stack in your app, is there a use case where this isn't trivial?

@peterwoodworth We hit a scenario where this became a real mess.

Given the example stack you referenced earlier here: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_certificatemanager-readme.html#cross-region-certificates

If I deploy that stack with a hostname (test.example.com) for my CloudFront distribution and Certificate, and then at a later stage, I need to rename my hostname (to test2.example.com) so I just update the properties.

Then when I try to deploy, it will fail with the following message:

0/4 Currently in progress: Stack1, Cert5C9FAEC1
Stack1 | 1/4 | 03:00:08 | UPDATE_COMPLETE      | AWS::CertificateManager::Certificate | Cert (Cert5C9FAEC1)
Stack1 | 1/4 | 03:00:10 | UPDATE_IN_PROGRESS   | Custom::CrossRegionExportWriter      | ExportsWritereuwest142AF533A/Resource/Default (ExportsWritereuwest142AF533A3E3B99E4)
Stack1 | 1/4 | 03:00:18 | UPDATE_FAILED        | Custom::CrossRegionExportWriter      | ExportsWritereuwest142AF533A/Resource/Default (ExportsWritereuwest142AF533A3E3B99E4) Received response status [FAILED] from custom resource. Message returned: Error: Some exports have changed!
/cdk/exports/Stack2/Stack1useast1RefCert5C9FAEC1924273F6
    at handler (/var/task/index.js:1:683)
    at Runtime.handler (/var/task/__entrypoint__.js:1:938)

The error message from here:

aws-cdk/packages/aws-cdk-lib/core/lib/custom-resource-provider/cross-region-export-providers/cross-region-ssm-writer-handler/index.ts

Line 26 in 06bea31

throw new Error('Some exports have changed!\n'+ changedExports.join('\n'));

indicates by the comment above that line that the export can never change, which would be a problem for certificates in the cases where you need to modify it (rename, add alternate names, etc).

[jalvinake]

If you are planning on deprecating DnsValidatedCertificate, please provide an alternative that we can use in our existing stacks. Moving our cert creating to a separate stack in us-east-1 and then doing cross-stack references is not a practical change. I wasted ~4 hours trying to make this work for an production web application today and have given up. Please reach out if you need more details.

[jcoppieters]

If you are planning on deprecating DnsValidatedCertificate, please provide an alternative that we can use in our existing stacks. Moving our cert creating to a separate stack in us-east-1 and then doing cross-stack references is not a practical change. I wasted ~4 hours trying to make this work for an production web application today and have given up. Please reach out if you need more details.

I couldn't agree more !

[vincent-dm]

I couldn't agree more !

Yes, me (and my colleagues) too! The current deprecation and dubious proposed migration path really throws AWS customers under the bus.

[blimmer]

I still think the best case scenario is that we get native region support in CloudFormation (see aws-cloudformation/cloudformation-coverage-roadmap#523 / #25343 (comment)). With that, this entire problem would go away.

If native support isn't possible before DnsValidatedCertificate is set to be removed in CDK v3, I think we should keep it. For my clients, at least, this issue could prevent us from upgrading.

[github-actions]

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

[olpeh]

We are also facing this issue but are unable to use the suggested 2 stack solution due to this:

>Error: Stack "stack2" cannot reference {stack1/Certificate/Resource[Ref]} in stack "stack1". Cross stack references are only supported for stacks deployed to the same account or between nested stacks and their parent stack

So, for us, this does not work:
#25343 (comment)

EDIT: never mind, we had forgotten to pass the account to the other stack and thus the error message was valid.

[metametadata]

There's a way to suppress annoying deprecation logs in Clojure/Java by filtering System/err: #24512 (comment).

[nickhudkins]

Could we please get an update on this issue?

[Gribbs]

this is madness.

[ruslan-roboto]

The necessity of having a dedicated cert stack and then using cross-region references has caused us a non-trivial amount of operational burden, and has been a time sink for developers. I understand that there is a lot on your plate and prioritization is more of an art than a science, but any update would be extremely helpful. Thanks!

[wparad]

It for sure is nonsensical for the suggestion to be create two stacks. Multiple stacks is almost always the wrong answer, it's totally unnecessary. Adds significant complexity and creates non-obvious problems. So we know that advice doesn't make sense. However, what is also a problem is that it isn't clear that there is a valid use case to continue to need create a us-east-1 certificate from another region.

The core of this problem comes from the fact that some resources can be deployed in any region through the AWS API, and other resources not. IAM and CloudFront come to mind. When you deploy these resources to AWS it doesn't matter which region you pick. However other resources that interact with them do require a region, it was always the case that certificates had to be in us-east-1 and lambda@edge functions have to be in us-east-1 and I believe there are others as well (I don't want to swear to it but things like WAF Global might also be the case).

I learned a long time ago two things here are 100% wrong:

• The AWS approach defined by the CloudFront API
• And, Because of AWS' approach here, deploying a CloudFront to a non-us-east-1 region

The correct solution is always deploying CloudFront in a us-east-1 region stack.

I believe the real issue is that some people have chosen to deploy CloudFront to a non-us-east-1 region stack, and now still need the cert in us-east-1. Before they had a solution, and now they also have a solution, but they don't like it.

There are 3 different solutions here:

• Recreate a new custom resource to deploy the us-east-1 certificate
• Create multiple stacks
• Migrate your resources to the us-east-1 region

As I pointed out (2) is wrong, and as @dickhardt pointed out (1) is also wrong. That leaves us with (3) the least wrong solution.

Because of the implementation AWS CloudFront is telling us to always use the us-east-1 region for our CloudFront stacks. I know that it is dumb and we don't want to do that, but asking the CDK to magically come to the rescue isn't the core problem. The CDK should help us make CloudFormation more bearable when it can, but the built-in custom resource was definitely a hack. Adding it back in is adding another hack ontop of the real problem. The reason the hack ever existed was to fix a completely different problem. The fact this supported the non-us-east-1 seems like merely a coincidence. So, in order for this to be fixed, the resources CloudFront is using would need to be changed. Of the owners of the CDK doesn't have the capability to make that happen.

I think we need to be fair to everyone here, There is no right answer, only wrong answers, because the fact you can deploy CloudFront outside of us-east-1 but not the certificate for it is cursed.

[dickhardt]

@wparad - I agree with most of what you say -- but the true solution would be to enable CloudFront certs to be created in the same region as the CloudFront Distribution -- just like you can for a load balancer or other services that can take a cert.

Making me create my CloudFront Distribution in us-east-1 complicates things even more as it then requires all the rest of my stack that sits behind the distribution to either be in a different region or in us-east-1.

No resources should have to be in us-east-1. Requiring that breaks inter region isolation -- a best practice.

Unfortunately fixing that is not out of scope for the CDK team, so it needs to get escalated up the chain -- in the meantime we continue with the hack.

[blimmer]

I said this above, but I still feel that the real/best solution is to push forward with upstream CloudFormation support: aws-cloudformation/cloudformation-coverage-roadmap#523. That would remove all the hacky workarounds in CDK and it would just work ™️.

Please upvote that issue and, if you pay for an enterprise support plan with AWS, bring it up with your TAMs.

[pausethelogic]

@dickhardt The main difference between CloudFront and other services like ALBs that can also use ACM certificates is that CloudFront isn't regional - it exists in the Global region. All "global" AWS services (CloudFront, Route53, etc) are run out of us-east-1 on the backend.

ACM certificates can only be used in the region where they were created, like the majority of regional AWS services. That's why ACM certificates must exist in us-east-1 to be used by CloudFront, the same way an ALB running out of us-west-2 must use an ACM certificate that also exists in us-west-2.

That being said, I can see how having a single construct that can run in any region and automatically create a certificate in us-east-1 if using CloudFront could be helpful for some CloudFormation/CDK customers.

It seems like the main issue is that the current construct abstracts away the fact that CloudFront isn't regional, so users didn't have to think about it when creating distributions.

[dickhardt]

@blimmer agree solving it in CloudFront would then also solve it for CDK.
@pausethelogic I was a PE in AWS ... I understand :) ... but since I can create a CF distribution in any region and manage it there, I should also be able to do the same with the needed certificate, which can be created in any region already.

[jk2l]

It for sure is nonsensical for the suggestion to be create two stacks. Multiple stacks is almost always the wrong answer, it's totally unnecessary. Adds significant complexity and creates non-obvious problems. So we know that advice doesn't make sense. However, what is also a problem is that it isn't clear that there is a valid use case to continue to need create a us-east-1 certificate from another region.

The core of this problem comes from the fact that some resources can be deployed in any region through the AWS API, and other resources not. IAM and CloudFront come to mind. When you deploy these resources to AWS it doesn't matter which region you pick. However other resources that interact with them do require a region, it was always the case that certificates had to be in us-east-1 and lambda@edge functions have to be in us-east-1 and I believe there are others as well (I don't want to swear to it but things like WAF Global might also be the case).

I learned a long time ago two things here are 100% wrong:

* The AWS approach defined by the CloudFront API

* And, Because of AWS' approach here, deploying a CloudFront to a non-us-east-1 region

The correct solution is always deploying CloudFront in a us-east-1 region stack.

I believe the real issue is that some people have chosen to deploy CloudFront to a non-us-east-1 region stack, and now still need the cert in us-east-1. Before they had a solution, and now they also have a solution, but they don't like it.

There are 3 different solutions here:

* Recreate a new custom resource to deploy the us-east-1 certificate

* Create multiple stacks

* Migrate your resources to the us-east-1 region

As I pointed out (2) is wrong, and as @dickhardt pointed out (1) is also wrong. That leaves us with (3) the least wrong solution.

Because of the implementation AWS CloudFront is telling us to always use the us-east-1 region for our CloudFront stacks. I know that it is dumb and we don't want to do that, but asking the CDK to magically come to the rescue isn't the core problem. The CDK should help us make CloudFormation more bearable when it can, but the built-in custom resource was definitely a hack. Adding it back in is adding another hack ontop of the real problem. The reason the hack ever existed was to fix a completely different problem. The fact this supported the non-us-east-1 seems like merely a coincidence. So, in order for this to be fixed, the resources CloudFront is using would need to be changed. Of the owners of the CDK doesn't have the capability to make that happen.

I think we need to be fair to everyone here, There is no right answer, only wrong answers, because the fact you can deploy CloudFront outside of us-east-1 but not the certificate for it is cursed.

The issue with this is the scope go beyond just CloudFront. Take Cognito custom domain as example.

Let say I want to create a Cognito in ap-southeast-2 (Sydney), with a custom domain (which also is a big requirement because you certain don't want your login domain like a random fraud domain).

Your application (ALB) is in ap-southeast-2. so, you HAVE TO deploy your Cognito, ALB both in ap-southeast-2. It sound straightforward so far, until you realise your custom domain ACM cert HAVE TO be create in us-east-1.

There is no simple single region stack solution here for Cognito custom domain.

Another example will be edge optimized API gateway, where you want your lambda run within same region as your application (e.g. in application VPC) you can deploy everything within any region but your ACM cert must be us-east-1 again.

and it is impossible to migrate my resources to us-east-1. (maybe i can do a cross region with transit gateway) but you are looking into bloat up your operation complexity for something that is much simpler to deploy within single region (also for compliance issue you maybe opening another can of worm)

like if you have PCI compliance where your network segregation is important to be as restrictive as possible

[jaecktec]

Having cross-domain stacks really doesn't cut it.

Imagine having this situation:

stack-a, eu-west-1:

• domain
• hosted zone

stack-b, us-east-1:

• certificate

stack-c, eu-west-1:

• cloudfront

now stack b needs information from stack a (domain name)
now stack c needs information from stack a and stack b (cert + domain name)

In addition:
auto-generated imports/exports are a pain to work with. As soon as you delete something you get into a deployment hell having to invert the pipeline by first deploying the dependent stack and then the origin, while for normal operation you deploy the origin first and then the dependent.

Not having this construct is a step backwads.

[dickhardt]

I spent some trying to make this happen cross region. What a mess.

I have S3 buckets in a different region from us-east-1, and my ECS are there as well. So all of those things need to get referenced by cloudfront.

I've figured out enough that I could probably create the certs in us-east-1 and then somehow get the ARNs to my other region, but what a mess.

Despite the party line that us-east-1 is fine -- I know enouth to know I won't deploy to us-east-1.

AWS needs to figure out how to enable cert management from any region. The same goes for Lambda@Edge.

[trautonen]

@gmarchand I already have this available in constructs hub:
DISCLAIMER: The construct is not officially supported by AWS or CDK and might be obsolete in the future if Cloudformation will support cross region/account operations
https://constructs.dev/packages/@trautonen/cdk-dns-validated-certificate/

The build is currently broken due to projen changes, but I will sort that out soon. The library itself should be working. Also has support for cross account zones with domain aliases.

[jaecktec]

@trautonen no offense to you, but I'm very paranoid when it comes to IaC libraries. I'd rather have this published by AWS.

Not meaning to say that you shouln't publish this, just want to mention that this is worse solution compared to what we have right now :(

[wparad]

@trautonen @gmarchand, while I appreciate both of you posting your solutions, anything that isn't officially blessed by the CDK team is going to struggle to find appropriate adoption. And given the main concern of AWS CFN not supporting the necessary functionality out of the box, your posts just encourage this issue to be a discussion on how to write a work around and not about encouraging the team to change the flaw in the service.

I would recommend removing your posts as to not encourage others to post their less sufficient solutions and turn this thread into a list of out of date/incomplete/inefficient solution posts.

[trautonen]

@wparad and @jaecktec, yea sure I understand your concerns and they are valid. Sadly AWS has taken their stance in this regard and cross region operations are just a hack or not working at all. My solution is loosely based on what was before in CDK, but as the CDK construct was fairly bad implementation I improved it to make it support more use cases than what CDK initially did. I will add a disclaimer to my message, but at least there's now a working solution that can be used instead of waiting AWS to sort out cross region resources properly.

[trautonen]

Also it's worth mentioning that in many projects I've done the root domain lies in separate account from the actual deployment account. There is no way in CDK to create certificates for those domains unless you use a custom resource. I guess the suggested AWS way of working around this is to manually create the certificates beforehand, but that defeats the purpose of automation.

So a common setup is to have domain delegation for subdomains like dev and prod, but then you also want to have the production available directly from the root domain like prod.mydomain.com and mydomain.com. Now you can't create certificates for those. My construct also solves this problem.

[mfittko]

Absolutely. It's kinda trivial to add a lambda backed CR using CDK, but the flaw is in the system. Why is route53 a global service and ACM not? It should be completely irrelanvant in which zone a cert is deployed.

[jk2l]

I think ideally we need this feature build in natively in Cfn as suggested in aws-cloudformation/cloudformation-coverage-roadmap#523

if this is supported natively then CDK don't need the custom resource anymore, everyone please give a thumb up in that issues as that's how AWS track the popularity of the feature request (as far as i know)

[blimmer]

Cross-posting @pahud's comment here: #9274 (comment)

Under the "Important Notes" subheading:

We acknowledge that the deprecation of DnsValidatedCertificate has caused friction, and we're working to improve the experience

I'm not sure what exactly that means, but maybe @pahud can clarify here.

My two cents: if CloudFormation can't or won't support this natively (aws-cloudformation/cloudformation-coverage-roadmap#523), I think DnsValidatedCertificate should be un-deprecated. The recommended approach of cross-region references is simply too complicated and, in many cases, causes other issues (like when a hosted zone is defined in a non-us-east-1 region and causes a circular dependency).