DeployCodeBaseCustomResource18AF4C97 failed

[TingDaoK]

❓ General Issue

Same code works fine couple weeks before, but without any code change in the few weeks, it starts to fail every time when I tried to do cdk deploy. The error reported is from DeployCodeBaseCustomResource18AF4C97. Seems like the underlying aws-cli command failed for some reason.

 47/49 | 9:47:38 AM | CREATE_FAILED        | Custom::CDKBucketDeployment           | DeployCodeBase/CustomResource/Default (DeployCodeBaseCustomResource18AF4C97) Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'cp', 's3://cdktoolkit-stagingbucket-574qtme5jv7w/assets/0b887894f0ecc543780dbe893f7115fc6ee8a5fde4ff9b9379406101a783a932.zip', '/tmp/tmplgzi9ntg/6f22d812-f3c1-434d-914d-585395b11860']' returned non-zero exit status 1. (RequestId: 852b0525-3088-4246-984e-221c86e5958f)

We used CDK for nodejs at version 1.103.0. The only related code I can think of is from @aws-cdk/aws-s3

        const code_bucket = new s3.Bucket(this, 'CodeBucket', {
            // removalPolicy: cdk.RemovalPolicy.DESTROY,
            // autoDeleteObjects: true,
        });
        new s3deploy.BucketDeployment(this, 'DeployCodeBase', {
            sources: [s3deploy.Source.asset('../benchmarks-stack')],
            destinationBucket: code_bucket,
        });

The whole CDK code can be found from https://github.com/awslabs/aws-c-s3/tree/main/benchmarks/dashboard-stack

Not sure it's bug from CDK or something in our code base.

The Question

Environment

• CDK CLI Version: 1.103.0
• Module Version: 1.103.0
• Node.js Version: v16.1.0
• OS: macOS
• Language (Version): TypeScript

Other information

[peterwoodworth]

This bucket is showing up in the error: s3://cdktoolkit-stagingbucket. This is the bucket created through bootstrapping. Have you modified that bucket at all?

[TingDaoK]

I don't think I ever touch that bucket, but it's on our team account, maybe someone else from my team did something. And I just checked that bucket seems like some testing file has been uploaded into it that makes sense. How should I fix it?

[TingDaoK]

Oh, the testing files are uploaded from our S3 test. So, it's there when our CDK script was still working a week before, but I can try to re-bootstrap it and see if it's the root cause of the failure.

Any suggestion about how to re-bootstrap the region? Just delete the bucket? I cannot find it from the documentation.

[peterwoodworth]

I suspect a re-bootstrap would solve this issue. I just cloned the repo and successfully deployed 😄

I'll post guidance on re-bootstrapping in a bit

[peterwoodworth]

@TingDaoK do you have newStyleStackSynthesis enabled? What happens when you try to bootstrap?

[TingDaoK]

Nope. But I also tried to add "@aws-cdk/core:newStyleStackSynthesis": true to cdk.json, but get the same result shown below:

dengket@3c22fb0f4430 dashboard-stack % cdk bootstrap aws://<account-id>/us-west-2
 ⏳  Bootstrapping environment aws://<account-id>/us-west-2...
 ✅  Environment aws://<account-id>/us-west-2 bootstrapped (no changes).

[peterwoodworth]

@TingDaoK it doesn't look like there's anything wrong with destroying the bootstrap stack. Unless you know of a specific reason not to destroy for your purposes, I'm not going to advise against destroying and recreating the bootstrap stack.

Here's how to do it

[TingDaoK]

I deleted the bootstrap stack and recreated one. However cdk delpoy still hang for ~10 minus and report the same error.
I also tried to deploy on my AWS account instead of our team account, the same error still.

[TingDaoK]

 51/53 | 10:16:55 AM | CREATE_COMPLETE      | AWS::EC2::Route                       | VPC/PrivateSubnet3/DefaultRoute (VPCPrivateSubnet3DefaultRoute27F311AE) 
Stack DashboardStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
51/53 Currently in progress: DeployCodeBaseCustomResource18AF4C97
Stack DashboardStack has an ongoing operation in progress and is not stable (CREATE_IN_PROGRESS)
 51/53 | 10:37:46 AM | CREATE_IN_PROGRESS   | Custom::CDKBucketDeployment           | DeployCodeBase/CustomResource/Default (DeployCodeBaseCustomResource18AF4C97) Resource creation Initiated
 51/53 | 10:37:46 AM | CREATE_FAILED        | Custom::CDKBucketDeployment           | DeployCodeBase/CustomResource/Default (DeployCodeBaseCustomResource18AF4C97) Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'cp', 's3://cdktoolkit-stagingbucket-r6i5bckjk009/assets/70c20a1ff55dff90648de2e1fa744f66c50e7aaa23eb5e2329c242bbe949f958.zip', '/tmp/tmp_fharugf/eb5ac530-85ec-4e64-aef4-03c53633f99f']' died with <Signals.SIGKILL: 9>. (RequestId: cfdd38cb-eae3-4a7b-8416-5f45a3d4bbbc)
        new CustomResource (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/@aws-cdk/core/lib/custom-resource.ts:119:21)
        \_ new BucketDeployment (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/@aws-cdk/aws-s3-deployment/lib/bucket-deployment.ts:224:5)
        \_ new DashboardStack (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/lib/dashboard-stack.ts:214:9)
        \_ Object.<anonymous> (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/bin/benchmarks.ts:15:1)
        \_ Module._compile (node:internal/modules/cjs/loader:1109:14)
        \_ Module.m._compile (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/ts-node/src/index.ts:858:23)
        \_ Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
        \_ Object.require.extensions.<computed> [as .ts] (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/ts-node/src/index.ts:861:12)
        \_ Module.load (node:internal/modules/cjs/loader:989:32)
        \_ Function.Module._load (node:internal/modules/cjs/loader:829:14)
        \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
        \_ main (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/ts-node/src/bin.ts:227:14)
        \_ Object.<anonymous> (/Users/dengket/project/aws-c-xx/aws-c-s3/benchmarks/dashboard-stack/node_modules/ts-node/src/bin.ts:513:3)
        \_ Module._compile (node:internal/modules/cjs/loader:1109:14)
        \_ Object.Module._extensions..js (node:internal/modules/cjs/loader:1138:10)
        \_ Module.load (node:internal/modules/cjs/loader:989:32)
        \_ Function.Module._load (node:internal/modules/cjs/loader:829:14)
        \_ Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
        \_ node:internal/main/run_main_module:17:47

[TingDaoK]

I also tried the CLI command reported failed separately, and it works fine.

dengket@3c22fb0f4430 dashboard-stack % aws s3 cp s3://cdktoolkit-stagingbucket-r6i5bckjk009/assets/70c20a1ff55dff90648de2e1fa744f66c50e7aaa23eb5e2329c242bbe949f958.zip /tmp/tmp_fharugf/eb5ac530-85ec-4e64-aef4-03c53633f99f
download: s3://cdktoolkit-stagingbucket-r6i5bckjk009/assets/70c20a1ff55dff90648de2e1fa744f66c50e7aaa23eb5e2329c242bbe949f958.zip to ../../../../../../../tmp/tmp_fharugf/eb5ac530-85ec-4e64-aef4-03c53633f99f

[nija-at]

Marking this as a bug for investigation.

[otaviomacedo]

Hi, @TingDaoK

I can't reproduce the issue, either, but can you try to increase the memory limit of the BucketDeployment:

new s3deploy.BucketDeployment(this, 'DeployCodeBase', {
    sources: [s3deploy.Source.asset('../benchmarks-stack')],
    destinationBucket: code_bucket,
    memoryLimit: 256 // default is 128
});

[TingDaoK]

Hi,

I just tested with the change, but still got the exactly same error.😥 So, you cannot reproduce the issue means you pulled the code base and successfully deployed the stack? Which platform/OS you used? Or is it related to the AWS account setting?

[peterwoodworth]

I'm using MacOS, and I pulled the code base and successfully deployed on my own AWS account just fine

[otaviomacedo]

Hi,

I just tested with the change, but still got the exactly same error.😥 So, you cannot reproduce the issue means you pulled the code base and successfully deployed the stack? Which platform/OS you used? Or is it related to the AWS account setting?

Yes, I suspect it's related to the AWS account setting. More specifically with the lambda function, which probably doesn't have enough memory or is timing out or something like that.

[fasatrix]

Suddenly I am experiencing the same thing. Assets are no longer transferred into the bucket from my windows machine and we have not changed anything. As a doublecheck we have also checkout the working code that was originally deployed 4 weeks ago and now it is no longer transferring the files

[TingDaoK]

The error reported is not informative. I don't know what settings to be changed. And as I saw, the lambda function is created by CDK as well. What kind of settings should I change?

[TingDaoK]

I checked the lambda function for CustomCDKBucketDeployment, its configured memory is 256 MB, and timeout is 15 mins. The code I want to deployed is around 500 KB. But still the deployment timed out. And my experience is the same as @fasatrix The same code works fine before and after several weeks without any change, it just fails now.

[TingDaoK]

Hmmm, I just run it again, now it deployed successfully.
I think it's very likely that I built the code to deploy locally and I forgot to delete node_modules folder and makes the code too big to deploy with the limited memory?

[fasatrix]

@TingDaoK you might want to take a look at the following issue. At least in our case is very close issue, so basically cdk.out contains cashed assets (.cache) which is causing the problem of releasing the previous asset all times and not the current created.
#12536

Our problem is exactly the same as this
https://www.rehanvdm.com/blog/cdk-shorts-1-consistent-asset-hashing-nodejs

[otaviomacedo]

@TingDaoK can you confirm whether the problem is related to cached assets?

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[fasatrix]

which is cau

By the transfer the creation of the assets in a lambda function... much more reliable

[omishagupta]

I was using KMS to upload the folder since I was uploading in encrypted bucket and was getting same error, while memory limit could have been one of the issues, it turned out despite of running the scripts as admin, the role lambda is using needs to be whitelisted on KMS too. Did that by using following snippet.

// Mention role to be used by bucket deployment
    new s3deploy.BucketDeployment(this, 'DeployWebsite', {
      sources: [s3deploy.Source.asset('../test')],
      destinationBucket: bucket,
      destinationKeyPrefix: props.folder,
      serverSideEncryption: s3deploy.ServerSideEncryption.AWS_KMS,
      serverSideEncryptionAwsKmsKeyId: props.key.keyArn,
      memoryLimit: 256, // as suggested here
      role: iam.Role.fromRoleArn(this, 'custom-role', <role-arn>)
    });
    
---
// Whitelist the role here
        const role = iam.Role.fromRoleArn(this, <construct-name>, <role-arn>)
        this.kmsKey.grantEncryptDecrypt(role)