Use of KMS keys across nested stacks create circular dependencies

[cmckni3]

I have a separate nested stack that creates and associates security related resources including IAM resources and KMS CMKs. Let's refer to this nested stack as SecurityNestedStack.

I have a nested stack which contains lambdas and SQS queues. Let's refer to this nested stack as LambdaNestedStack.

These queues are using KMS CMKs and using an IAM role from SecurityNestedStack. Everything for the role and cmk has been preconfigured in SecurityNestedStack including policies for SQS IAM access, KMS IAM access, and CMK key policy.

Setting up the SQSEventSource in LambdaNestedStack calls grantConsumeMessages which in turn adds two new IAM policies to the queue's IAM role. This happens in the wrong stack and causes a circular dependency.

This is helpful for ease of setting up events, roles, and key policies for the queue. It's problematic in more advanced use cases and larger stacks that must be split up (200 resource limit in stacks).

Reproduction Steps

See above. I can post code if necessary.

Error Log

CloudFormation reports that all nested stacks have circular dependencies. Not completely true but there is a circular dependency across nested stacks.

Environment

• CLI Version : 1.19.0
• Framework Version: 1.19.0
• OS : macOS
• Language : TypeScript

Other

My workaround right now is to call addEventSourceMapping and pass in batchSize and eventSourceArn myself OR import the role.

---

This is 🐛 Bug Report

[cmckni3]

This is probably a general problem but I gave my example using lambda and SQS.

#4356 describes a situation with SQS and API gateway. I believe this will apply across the board with builtin implementations of IEventSource (S3EventSource, SNSEventSource, SQSEventSource, DynamoEventSource, etc)

[nija-at]

A reproduction code here would help a lot in the diagnosis.

I've tried reproducing this myself but was not able to. Here's the code I used -

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from '@aws-cdk/core';
import { NestedStack } from '@aws-cdk/aws-cloudformation';
import { Construct, Stack } from '@aws-cdk/core';
import { Queue } from '@aws-cdk/aws-sqs';
import { Function, Code, Runtime } from '@aws-cdk/aws-lambda';
import { Key } from '@aws-cdk/aws-kms';
import { SqsEventSource } from '@aws-cdk/aws-lambda-event-sources'; 

class FirstNestedStack extends NestedStack {
  public readonly key: Key;

  constructor(scope: Construct, id: string) {
    super(scope, id);

    this.key = new Key(this, 'key');
  }
}

class SecondNestedStack extends NestedStack {
  constructor(scope: Construct, id: string, firstStack: FirstNestedStack) {
    super(scope, id);

    const queue = new Queue(this, 'queue', {
      encryptionMasterKey: firstStack.key,
    });

    const fn = new Function(this, 'function', {
      code: Code.fromInline('foo'),
      runtime: Runtime.NODEJS_10_X,
      handler: 'index.handler'
    });

    fn.addEventSource(new SqsEventSource(queue));
  }
}

class MainStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    const firstStack = new FirstNestedStack(this, 'FirstNestedStack');
    new SecondNestedStack(this, 'SecondNestedStack', firstStack);
  }
}

const app = new cdk.App();
new MainStack(app, 'MainStack');

I was able to synthesize using cdk synth and a cursory look over the synthesized stacks showed everything looked good.

[nija-at]

@cmckni3 - I copied that over and added the two lines below -

const app = new App();
new MainStack(app, 'MainStack');

cdk synth works just fine and prints the output as expected. I don't see any circular dependency errors.

I'm using cdk version 1.21.0, but I don't see why it wouldn't work for 1.19.0.

[cmckni3]

Error happens on cdk deploy. The error comes from Cloudformation validation.

[nija-at]

Ah I see.

I can confirm this occurs even for the repro code I had added in my original comment here - #5765 (comment) - and occurs only during cdk deploy and not cdk synth.

@cmckni3 - would have loved for all of this to be clearer in the issue description under the section 'reproduction steps'. That section is dedicated to be filled in with such information.

[nija-at]

Here's the smallest code needed to reproduce -

import { App, Construct, Stack } from '@aws-cdk/core';
import { NestedStack } from '@aws-cdk/aws-cloudformation';
import { Key } from '@aws-cdk/aws-kms';
import { Code, Function, Runtime } from '@aws-cdk/aws-lambda';

export class MainStack extends Stack {
  constructor(scope: App, id: string) {
    super(scope, id);

    const firstStack = new FirstNestedStack(this, 'FirstNestedStack');
    new SecondNestedStack(this, 'SecondNestedStack', firstStack);
  }
}

class FirstNestedStack extends NestedStack {
  public readonly key: Key;

  constructor(scope: Construct, id: string) {
    super(scope, id);

    this.key = new Key(this, 'Key');
  }
}

class SecondNestedStack extends NestedStack {
  constructor(scope: Construct, id: string, firstStack: FirstNestedStack) {
    super(scope, id);

    const fn = new Function(this, 'function', {
      code: Code.fromInline('foo'),
      runtime: Runtime.NODEJS_10_X,
      handler: 'index.handler'
    });

    firstStack.key.grantDecrypt(fn);
  }
}

const app = new App();
new MainStack(app, 'MainStack');

On further debugging, the issue isn't with SQSEventSource, it's got to do with KMS.

Since the key is defined in FirstStack but used in the SecondStack, there's a dependency created where SecondStack depends on FirstStack
Next, the call to KMS key grantDecrypt now seems to create a dependency where the FirstStack depends on the SecondStack.

I suspect that the code somewhere in the grant method doesn't account for nested stacks.

cc @skinny85 who is the last major change to this area of code.

[cmckni3]

Ok, I felt like the description and error log covered it. Guess I didn’t call out cdk deploy specifically.

CloudFormation reports that all nested stacks have circular dependencies. Not completely true but there is a circular dependency across nested stacks.

[abbottdev]

We're also hitting this problem - but with a slightly different use case - we're trying to define our KMS keys in a Stack for our SNS/SQS resources; we then have an ECS Stack which contains other resources. We're trying to use the QueueProcessingFargateService construct; however this does a similar grant See here:

aws-cdk/packages/@aws-cdk/aws-ecs-patterns/lib/base/queue-processing-service-base.ts

Line 312 in e8801a0

this.sqsQueue.grantConsumeMessages(service.taskDefinition.taskRole);

This happens when trying to use an existing queue as a prop passed in to the ECS task. This causes the same circular dependency as provided in the simple example.

[Mickael-van-der-Beek]

Any updates on this issue? This seems to be an ongoing problem.