Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(s3): (CustomCDKBucketDeployment custom resource Lambda function fails when updating Bucket id) #27660

Open
massi-ang opened this issue Oct 24, 2023 · 2 comments
Open

(s3): (CustomCDKBucketDeployment custom resource Lambda function fails when updating Bucket id) #27660

massi-ang opened this issue Oct 24, 2023 · 2 comments
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. effort/small Small work item – less than a day of effort p2

Comments

@massi-ang
Copy link

massi-ang commented Oct 24, 2023 •
edited
Loading

Describe the bug

I have a stack using a BucketDeployment resource configured with retainOnDelete: false using an S3 Bucket configured with autoDeleteObjects: true.

When updating the s3.Bucket resource id and redeploying, the stack fails with the following error:

10:10:46 AM | UPDATE_FAILED        | Custom::CDKBucketDeployment | stack1CustomResource4003511E
Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deployst
ack-bucket43879c71-1hhkjuylg244r/', '--recursive']' returned non-zero exit status 1. (RequestId: 65867ce6-f4d3-4c8e-bf8e-46dacee629
54)

Expected Behavior

No errors are thrown

Current Behavior

The following error is thrown:

10:10:46 AM | UPDATE_FAILED        | Custom::CDKBucketDeployment | stack1CustomResource4003511E
Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deployst
ack-bucket43879c71-1hhkjuylg244r/', '--recursive']' returned non-zero exit status 1. (RequestId: 65867ce6-f4d3-4c8e-bf8e-46dacee629
54)

The logs for the CDKBucketDeployment Lambda functions are as follow:

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                message                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1698137767364 | INIT_START Runtime Version: python:3.9.v34 Runtime Version ARN: arn:aws:lambda:eu-west-1::runtime:25730c9630b4654f0753a405e39f452e48dad54fbcae4ce598d051c13d109bf8                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1698137767639 | [INFO] 2023-10-24T08:56:07.639Z  Found credentials in environment variables.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1698137767798 | START RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671 Version: $LATEST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698137767798 | [INFO] 2023-10-24T08:56:07.798Z 7fed9806-b5ad-438c-8f64-4eab012fa671 {'RequestType': 'Create', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': '0e3b941c-7fbf-4e0c-94ae-92b06815383d', 'LogicalResourceId': 'stack1CustomResource4003511E', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | old_s3_dest: None                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| 1698137767799 | [INFO] 2023-10-24T08:56:07.799Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | workdir: /tmp/tmpfsiuz_zo                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1698137767817 | [INFO] 2023-10-24T08:56:07.816Z 7fed9806-b5ad-438c-8f64-4eab012fa671 archive: /tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698137767817 | [INFO] 2023-10-24T08:56:07.817Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | aws s3 cp s3://cdk-hnb659fds-assets-699391019698-eu-west-1/8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip /tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1698137787617 | Completed 2.7 KiB/2.7 KiB (9.8 KiB/s) with 1 file(s) remaining download: s3://cdk-hnb659fds-assets-699391019698-eu-west-1/8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip to ../../tmp/tmpfsiuz_zo/b8c519a1-28bd-4546-912e-43a04d153982                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1698137789217 | [INFO] 2023-10-24T08:56:29.217Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | extracting archive to: /tmp/tmpfsiuz_zo/contents                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| 1698137789217 | [INFO] 2023-10-24T08:56:29.217Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | markers: {}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1698137789238 | [INFO] 2023-10-24T08:56:29.238Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | aws s3 sync --delete /tmp/tmpfsiuz_zo/contents s3://s3deploystack-bucket43879c71-efwtvnkgutvp/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| 1698137806037 | Completed 2.8 KiB/3.8 KiB (17.3 KiB/s) with 3 file(s) remaining upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.js to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.js                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1698137806377 | Completed 2.8 KiB/3.8 KiB (17.3 KiB/s) with 2 file(s) remaining Completed 2.8 KiB/3.8 KiB (5.5 KiB/s) with 2 file(s) remaining  upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.d.ts to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.d.ts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698137806417 | Completed 2.8 KiB/3.8 KiB (5.5 KiB/s) with 1 file(s) remaining Completed 3.8 KiB/3.8 KiB (6.8 KiB/s) with 1 file(s) remaining upload: ../../tmp/tmpfsiuz_zo/contents/s3-deploy.ts to s3://s3deploystack-bucket43879c71-efwtvnkgutvp/s3-deploy.ts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1698137807980 | [INFO] 2023-10-24T08:56:47.980Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | response body: {"Status": "SUCCESS", "Reason": "See the details in CloudWatch Log Stream: 2023/10/24/[$LATEST]0ed22de7de3b4371b62cb69ec2abf684", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "0e3b941c-7fbf-4e0c-94ae-92b06815383d", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {"DestinationBucketArn": null, "SourceObjectKeys": ["8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip"]}}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1698137808352 | [INFO] 2023-10-24T08:56:48.352Z 7fed9806-b5ad-438c-8f64-4eab012fa671 | status code: OK                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698137808359 | END RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698137808359 | REPORT RequestId: 7fed9806-b5ad-438c-8f64-4eab012fa671 Duration: 40560.05 ms Billed Duration: 40561 ms Memory Size: 128 MB Max Memory Used: 128 MB Init Duration: 432.95 ms                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| 1698138013699 | START RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Version: $LATEST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f {'RequestType': 'Update', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': 'f999927a-bc46-4be2-aebf-4897423dd642', 'LogicalResourceId': 'stack1CustomResource4003511E', 'PhysicalResourceId': 'aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket136091df3-fmpx07qm4scw', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}, 'OldResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}}  |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | s3_dest: s3://s3deploystack-bucket136091df3-fmpx07qm4scw/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | old_s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698138013700 | [INFO] 2023-10-24T09:00:13.700Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | aws s3 rm s3://s3deploystack-bucket43879c71-efwtvnkgutvp/ --recursive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| 1698138030279 | fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| 1698138031879 | [ERROR] 2023-10-24T09:00:31.859Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1. Traceback (most recent call last):   File "/var/task/index.py", line 115, in handler     aws_command("s3", "rm", old_s3_dest, "--recursive")   File "/var/task/index.py", line 233, in aws_command     subprocess.check_call([aws] + list(args))   File "/var/lang/lib/python3.9/subprocess.py", line 373, in check_call     raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1698138031879 | [ERROR] 2023-10-24T09:00:31.879Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | cfn_error: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1698138031880 | [INFO] 2023-10-24T09:00:31.880Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | response body: {"Status": "FAILED", "Reason": "Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket43879c71-efwtvnkgutvp/', '--recursive']' returned non-zero exit status 1.", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "f999927a-bc46-4be2-aebf-4897423dd642", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {}}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698138032124 | [INFO] 2023-10-24T09:00:32.124Z bb8ea787-c893-4e8e-ad49-6dfc128d4d6f | status code: OK                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698138032159 | END RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698138032159 | REPORT RequestId: bb8ea787-c893-4e8e-ad49-6dfc128d4d6f Duration: 18459.66 ms Billed Duration: 18460 ms Memory Size: 128 MB Max Memory Used: 128 MB                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| 1698138055127 | START RequestId: b698df79-1457-4fdf-9c1e-5555b327c105 Version: $LATEST                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698138055127 | [INFO] 2023-10-24T09:00:55.127Z b698df79-1457-4fdf-9c1e-5555b327c105 {'RequestType': 'Update', 'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'StackId': 'arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf', 'RequestId': '85ff64d4-c59b-425b-963e-8d85ca9e9191', 'LogicalResourceId': 'stack1CustomResource4003511E', 'PhysicalResourceId': 'aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket43879c71-efwtvnkgutvp', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}, 'OldResourceProperties': {'ServiceToken': 'arn:aws:lambda:eu-west-1:699391019698:function:S3DeployStack-CustomCDKBucketDeployment8693BB64968-A4olc2eGAizr', 'RetainOnDelete': 'false', 'Prune': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-699391019698-eu-west-1'], 'DestinationBucketName': 's3deploystack-bucket136091df3-fmpx07qm4scw', 'SourceObjectKeys': ['8266d90f1103990f66552d57026611ebc7a9909ee7f7fd5d33d71f181ffb5a25.zip']}}  |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | s3_dest: s3://s3deploystack-bucket43879c71-efwtvnkgutvp/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | old_s3_dest: s3://s3deploystack-bucket136091df3-fmpx07qm4scw/                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1698138055128 | [INFO] 2023-10-24T09:00:55.128Z b698df79-1457-4fdf-9c1e-5555b327c105 | aws s3 rm s3://s3deploystack-bucket136091df3-fmpx07qm4scw/ --recursive                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| 1698138071759 | fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| 1698138073399 | [ERROR] 2023-10-24T09:01:13.399Z b698df79-1457-4fdf-9c1e-5555b327c105 Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1. Traceback (most recent call last):   File "/var/task/index.py", line 115, in handler     aws_command("s3", "rm", old_s3_dest, "--recursive")   File "/var/task/index.py", line 233, in aws_command     subprocess.check_call([aws] + list(args))   File "/var/lang/lib/python3.9/subprocess.py", line 373, in check_call     raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1698138073399 | [ERROR] 2023-10-24T09:01:13.399Z b698df79-1457-4fdf-9c1e-5555b327c105 | cfn_error: Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| 1698138073400 | [INFO] 2023-10-24T09:01:13.400Z b698df79-1457-4fdf-9c1e-5555b327c105 | response body: {"Status": "FAILED", "Reason": "Command '['/opt/awscli/aws', 's3', 'rm', 's3://s3deploystack-bucket136091df3-fmpx07qm4scw/', '--recursive']' returned non-zero exit status 1.", "PhysicalResourceId": "aws.cdk.s3deployment.a6226748-00ba-435a-a397-1930e4c610a0", "StackId": "arn:aws:cloudformation:eu-west-1:699391019698:stack/S3DeployStack/0288af90-724b-11ee-af8f-0673370ccdbf", "RequestId": "85ff64d4-c59b-425b-963e-8d85ca9e9191", "LogicalResourceId": "stack1CustomResource4003511E", "NoEcho": false, "Data": {}}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| 1698138073661 | [INFO] 2023-10-24T09:01:13.661Z b698df79-1457-4fdf-9c1e-5555b327c105 | status code: OK                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| 1698138073699 | END RequestId: b698df79-1457-4fdf-9c1e-5555b327c105                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| 1698138073699 | REPORT RequestId: b698df79-1457-4fdf-9c1e-5555b327c105 Duration: 18572.32 ms Billed Duration: 18573 ms Memory Size: 128 MB Max Memory Used: 128 MB                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Reproduction Steps

Create a new CDK app, and copy the following stack code:

import * as cdk from 'aws-cdk-lib';
import { BlockPublicAccess, Bucket } from 'aws-cdk-lib/aws-s3';
import { BucketDeployment, Source } from 'aws-cdk-lib/aws-s3-deployment';
import { Construct } from 'constructs';
import * as path from 'path';

export class S3DeployStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const bucket = new Bucket(this, "bucket", {
      autoDeleteObjects: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      blockPublicAccess: BlockPublicAccess.BLOCK_ALL,
    });
   
    const s3deploy = new BucketDeployment(this, 'stack-1', {
      destinationBucket: bucket, 
      sources: [Source.asset(path.join(__dirname, "../bin"))],
      retainOnDelete: false,
    })
  }
}

Compile and deploy.

Next, change the Bucket resource id and redeploy.

The error is thrown and the stack goes into a failed rollback state from which one can only recover by destroying the stack.

Possible Solution

Additional Information/Context

No response

CDK CLI Version

2.99.1 (build b2a895e)

Framework Version

2.99.1

Node.js Version

v20.2.0

OS

MacOS

Language

TypeScript

Language Version

5.1.3

Other information

No response
@massi-ang massi-ang added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 24, 2023
@github-actions github-actions bot added the @aws-cdk/aws-s3 Related to Amazon S3 label Oct 24, 2023
@massi-ang massi-ang mentioned this issue Oct 24, 2023
Closed
@massi-ang massi-ang changed the title (s3): (S3AutoDelete custom resource Lambda function role lacks necessary permissions) (s3): (S3AutoDelete custom resource Lambda function fails when updating Bucket id) Oct 24, 2023
@massi-ang massi-ang changed the title (s3): (S3AutoDelete custom resource Lambda function fails when updating Bucket id) (s3): (CustomCDKBucketDeployment custom resource Lambda function fails when updating Bucket id) Oct 24, 2023
@khushail khushail added investigating This issue is being investigated and/or work is in progress to resolve the issue. and removed needs-triage This issue or PR still needs to be triaged. labels Oct 24, 2023
@khushail khushail self-assigned this Oct 24, 2023
@khushail
Copy link
Contributor

@massi-ang thanks for reporting this. I am able to reproduce this with the provided code.

@khushail khushail added p2 needs-review effort/small Small work item – less than a day of effort and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. labels Oct 25, 2023
@khushail khushail removed their assignment Oct 25, 2023
@github-actions github-actions bot mentioned this issue Nov 1, 2023
Closed
@huantbui
Copy link

huantbui commented Mar 1, 2024 •
edited
Loading

Hello, I had a similar issue and behind the scene, BucketDeployment will create a Lambda (python) that will run awscdli, s3, sync --delete and the CDK would show this error:

Received response status [FAILED] from custom resource. Message returned: Command '['/opt/awscli/aws', 's3', 'sync', '--delete', '/tmp//contents', 's3://, ...]' returned non-zero exit status 1.

However, i went to look at the Lambda that the BucketDeployment with named function: *-CustomCDKBucketDeployment and look at the logs, you will see this error before that error shown above

An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

I then went in and updated my s3.PolicyStatement to allow specific actions: PutObject*, DeleteObject* - and it works...

I hope that will work for you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/aws-s3 Related to Amazon S3 bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@huantbui @massi-ang @khushail