Custom::LogRetention - ThrottlingException when creating multiple resources

[shantgup]

Describe the bug

CDK,

Creating this on behalf of an enterprise customer that experienced significant pains & delays due to this issue.

The Custom::LogRetention resource backed by a Lambda function managed by CDK with the source code here, gets throttled by CloudWatch Logs when multiple resources are being created.

Recommend implementing backoff and retry mechanism on this custom resource to prevent failure due to throttling.

Expected Behavior

The Custom::LogRetention resource should backoff and retry when it encounters a ThrottlingException.

Current Behavior

Currently, the Custom::LogRetention resource fails when it encounters a ThrottlingException, and this causes the entire CloudFormation stack to fail and rollback.

Reproduction Steps

Deploy any CloudFormation stack with multiple log-retention-provider that in turn create Custom::LogRetention resources.

Possible Solution

Implement backoff and retry until successful.

Additional Information/Context

No response

CDK CLI Version

2.67.0

Framework Version

No response

Node.js Version

14.x

OS

Lambda

Language

Typescript

Language Version

No response

Other information

No response

[khushail]

Hi @shantgup , thanks for reaching out. This LogRetentionRetryOptions might be helpful to go through for what you are looking for.

Could you also share the full error messages on the CDK deployment as well ?
We actually had LogRetentionRetryOptions that was previously designed to work around a OperationAbortedException error(see #17546). Behind the scene, log-retention-provider will just creates the log groups with SDK retry options and handle the ResourceAlreadyExistsException and OperationAbortedException. Not sure if we should include ThrottlingException here but having full error messages would be very helpful for us to address this issue.

Let me know if this does not help.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.