Amplify cdk-stack output missing dependencies which are generated during cdk synth

[joekiller]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v16.20.0

Amplify CLI Version

12.0.3

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

The way amplify builds a cdk stack and then generates cfn is different than the new cdk synthesize and therefore doesn't "satisfy missing context" as the synthesize function mentions. Due to this difference stacks created in amplify fail to render more complex dependency trees. The easiest example of this is when an lambda event source is mapped. The resulting policy dependency isn't applied as it should and the deploy fails. This different in output explains the reason that items like #10332

If the generate cfn of amplify difference is applicable across the whole library, it likely explains why people experience stuff like #10395.

Here is an example output:

CREATE_FAILED
Resource handler returned message: "Invalid request provided: The provided execution role does not have permissions to call Publish on SNS (Service: Lambda, Status Code: 400, Request ID: 0" (RequestToken: 4, HandlerErrorCode: InvalidRequest)

Expected behavior

I expect amplify to compile the cdk templates correctly.

Reproduction steps

The easiest way is to create a lambda event source binding.

1. Create an amplify stack.
2. Add custom cdk.
3. Add an event source binding.

...
 // Dynamo DB Tables
    const customTable = new Table(this, 'customTable', {
      partitionKey: { name: 'id', type: AttributeType.STRING},
      timeToLiveAttribute: 'expirationDate',
      tableName: Fn.join('', ['custom-table', Fn.ref('env')]),
      stream: StreamViewType.OLD_IMAGE,
      removalPolicy: RemovalPolicy.DESTROY,
      pointInTimeRecovery: true,
    });

    const reactToTableLambda = new Function(this, 'reactiveLambda', {
      handler: "index.handler",
      code: new InlineCode('return'),
      runtime: Runtime.NODEJS_16_X
    });

    // SNS topic for DLQ
    const snsTopic = new Topic(this, 'reactiveLambdaFail', {
      displayName: 'Reactive Lambda Fail',
    });

    // lambda event source
    reactToTableLambda.addEventSource(new DynamoEventSource(customTable, {
      startingPosition: StartingPosition.LATEST,
      retryAttempts: 3,
      onFailure: new SnsDlq(snsTopic),
    }));

Demonstation stack here.
amplify-scratch-cannot-deploy-this.zip which is a download of this repo: https://github.com/joekiller/amplify-scratch/tree/cannot-deploy-this

Project Identifier

196a15c2-6185-4f3b-9126-51b1260992f5

Log output

# Put your logs below this line

Additional information

Below is a screenshot of the dependency difference that results on the lambda resource.

I've created a cdk enabled workflow with the amplify export functionality at the following main branch to emulate deploying cdk-custom with amplify by removing the custom cdk stack from amplify proper and deploying the resources via cdk by attaching the cdk resources to the amplify exported resources after the fact. The custom resources are only synthesized by cdk. https://github.com/joekiller/amplify-scratch

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey @joekiller, thank you for reaching and apologies in the delay in a response. Tried reproducing the issue with the following steps.

1. created a new Amplify project and also a new cdk project.
2. in the Amplify project added a custom resource with the example code provided
3. pushed the resources and observed no errors
4. utilized amplify export to export the resources to the cdk project.
5. utilized AmplifyExportedBackend to use the exported backend
6. then deployed the cdk app
   but did not observe an error.

additionally, i did not observe an dependsOn in the generated cfn template

Could you provide us some additional information that would assist us in reproduction. Additionally, could you let us know the permissions on the aws profile that the CDK app utilizes.

[joekiller]

@ykethan please create an independent cdk project and add the same resource declarations to that project and do a cdk synth to observe the dependsOn declarations that are expected.

If you export the amplify project and use the amplify exported backend then, yes, you will not see the dependencies as that literally imports via CfnImport. The import amplify backend doesn't render the proper stack either as it is using CfnImport, a verbatim copy of what amplify created.

[joekiller]

You are correct that sometimes the stack will deploy as is because there's a race between the creation of the eventSource and attachment of the policy to the role. If you examine the resource creation order as cloud formation proceeds you will observe that the attachment of the role creation starts at the same time as creating the eventSource. The attachment of the policy should always precede the creation of the eventSource and to ensure the policy is always attached prior to the eventSource mapping one must make the lambda itself depend on the policy. Having the lambda depend on the policy ensures the policy statement is always created and attached to the role prior to creating the lambda which has the mapping. The lambda does have the role but the policy attachment to the role and eventSource is the race.

If you want, you can examine the SNS policy and see that it is attached to the role during the same create

[joekiller]

For anyone using my workaround please note that you must remove the custom resource from the Amplify stack and only define it in the cdk project. You can export all non-custom resources from amplify and then declare the cdk resources in a separate stack. My workaround attaches and synthesizes the custom stack in the cdk stack only.

[joekiller]

@ykethan when I created the example please observe that for the cdk stack i include the same code that the Amplify cli uses to synthesize the stack however I do not use the import cdk resource.

I just included the custom-cdk in the tsconfig so that the cdk application cdk/app/cdk.ts could synthesize the same stack that Amplify does. In my example the cdk stack doesn't use aws-amplify/cdk-exported-backend at all.

[ykethan]

Hey @joekiller, thank you for the information. I was able to reproduce the issue. It appears the Amplify CLI provided synth version is causing this behaviour. Marking this as bug for further improvements.

[webjay]

@ykethan I also have this problem, trying to use ContainerImage.fromAsset. Is there a way around it or a fix soon?