Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_appconfig_alpha: delete fail on stack destroy #26804

Closed
ran-isenberg opened this issue Aug 18, 2023 · 20 comments
Closed

aws_appconfig_alpha: delete fail on stack destroy #26804

ran-isenberg opened this issue Aug 18, 2023 · 20 comments
Labels
@aws-cdk/aws-appconfig Related to AWS AppConfig bug This issue is a bug. effort/small Small work item – less than a day of effort p2

Comments

@ran-isenberg
Copy link

Describe the bug

I'm using the new L2 constructs, sample code below.
Deploying works just fine.
However, on destroy i get the following error:
11:24:04 AM | DELETE_FAILED | AWS::AppConfig::ConfigurationProfile | rootappconfigOrder...ionProfile9C631413 Cannot delete configuration profile bvdne2p because there are still hosted configuration versions existing under it. (Service: AmazonAppConfig; Status Code: 400; Error Code: BadRequestException; Request ID: d1d449de-9138-4a5e-8461-05f925d07bae; Proxy: null)

Python code:

from pathlib import Path

import aws_cdk.aws_appconfig_alpha as appconfig
from aws_cdk import Duration
from constructs import Construct

from cdk.my_service.configuration.schema import FeatureFlagsConfiguration

DEFAULT_DEPLOYMENT_STRATEGY = 'AppConfig.AllAtOnce'

CUSTOM_ZERO_TIME_STRATEGY = 'zero'


class ConfigurationStore(Construct):

    def __init__(self, scope: Construct, id_: str, environment: str, service_name: str, configuration_name: str) -> None:
        """
        This construct should be deployed in a different repo and have its own pipeline so updates can be decoupled from
        running the service pipeline and without redeploying the service lambdas.

        Args:
            scope (Construct): The scope in which to define this construct.
            id_ (str): The scoped construct ID. Must be unique amongst siblings. If the ID includes a path separator (``/``), then it will be
                        replaced by double dash ``--``.
            environment (str): environment name. Used for loading the corresponding JSON file to upload under
                               'configuration/json/{environment}_configuration.json'
            service_name (str): application name.
            configuration_name (str): configuration name
        """
        super().__init__(scope, id_)

        configuration_str = 'bla bla freeform conf'
        self.app_name = f'{id_}{service_name}'
        self.config_app = appconfig.Application(
            self,
            id=self.app_name,
            name=self.app_name,
        )

        self.config_env = appconfig.Environment(
            self,
            id=f'{id_}env',
            application=self.config_app,
            name=environment,
        )

        # zero minutes, zero bake, 100 growth all at once
        self.config_dep_strategy = appconfig.DeploymentStrategy(
            self,
            f'{id_}{CUSTOM_ZERO_TIME_STRATEGY}',
            rollout_strategy=appconfig.RolloutStrategy.linear(
                growth_factor=100,
                deployment_duration=Duration.minutes(0),
                final_bake_time=Duration.minutes(0),
            ),
        )

        self.config = appconfig.HostedConfiguration(
            self,
            f'{id_}version',
            application=self.config_app,
            name=configuration_name,
            content=appconfig.ConfigurationContent.from_inline(configuration_str),
            type=appconfig.ConfigurationType.FREEFORM,
            content_type='application/json',
            deployment_strategy=self.config_dep_strategy,
            deploy_to=[self.config_env],
        )

You can see the changes I made from the CFN variation which worked just fine here:
https://github.com/ran-isenberg/aws-lambda-handler-cookbook/pull/710/files

Expected Behavior

Stack deletion success.

Current Behavior

I need to manually delete the resources via console.

root-appconfig-Orders: destroying... [1/1]
11:24:04 AM | DELETE_FAILED | AWS::AppConfig::ConfigurationProfile | rootappconfigOrder...ionProfile9C631413
Cannot delete configuration profile bvdne2p because there are still hosted configuration versions existing under it. (Service: AmazonAppConfig; Status Code: 400; Error Code:
BadRequestException; Request ID: d1d449de-9138-4a5e-8461-05f925d07bae; Proxy: null)
❌ root-appconfig-Orders: destroy failed Error: The stack named root-appconfig-Orders is in a failed state.

Reproduction Steps

Use the sample code I provided to create the AppConfig resources

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.92.0 (build bf62e55)

Framework Version

No response

Node.js Version

18.17.1

OS

Mac Ventura 13.5

Language

Python

Language Version

No response

Other information

No response

@ran-isenberg ran-isenberg added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2023
@github-actions github-actions bot added the @aws-cdk/aws-appconfig Related to AWS AppConfig label Aug 18, 2023
@schmiddy
Copy link
Contributor

Yes, we hit the same error when attempting to delete a stack using the new aws-appconfig-alpha. I am testing out a Custom Resource tied to the Application that will go through all the dependencies (including Hosted Config Versions) and delete them all when the Application is attempted to be deleted, it seems to work OK.

@ran-isenberg
Copy link
Author

TBH, I'd rather use the older CFNs instead of adding a custom resource.

@niebloomj
Copy link

Does this issue not occur in the non-alpha CFN version?

@ran-isenberg
Copy link
Author

Does this issue not occur in the non-alpha CFN version?

nope, CFN works great. You need define a lot more, but it works and gets deleted just fine.
see example here:
https://github.com/ran-isenberg/aws-lambda-handler-cookbook/blob/main/cdk/my_service/configuration/configuration_construct.py

@peterwoodworth
Copy link
Contributor

Could you please share both of the templates that get generated from before and after this PR @ran-isenberg? Additionally, could you note during the deletion of the stack, if the resources in those stacks get deleted in a different order?

@peterwoodworth peterwoodworth added p1 and removed needs-triage This issue or PR still needs to be triaged. labels Aug 18, 2023
@ran-isenberg
Copy link
Author

yes. Stack names are different due to branch names difference, but it is as you requested.
CFN version:

{
 "Resources": {
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfOrders91C8FCAD": {
   "Type": "AWS::AppConfig::Application",
   "Properties": {
    "Name": "root-main-Ordersdynamic_confOrders"
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confOrders"
   }
  },
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfenvE9D41676": {
   "Type": "AWS::AppConfig::Environment",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfOrders91C8FCAD"
    },
    "Name": "dev"
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confenv"
   }
  },
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfprofile6783FAA5": {
   "Type": "AWS::AppConfig::ConfigurationProfile",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfOrders91C8FCAD"
    },
    "LocationUri": "hosted",
    "Name": "my_conf"
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confprofile"
   }
  },
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfversion93674A9E": {
   "Type": "AWS::AppConfig::HostedConfigurationVersion",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfOrders91C8FCAD"
    },
    "ConfigurationProfileId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfprofile6783FAA5"
    },
    "Content": "{\n    \"features\": {\n        \"premium_features\": {\n            \"default\": false,\n            \"rules\": {\n                \"enable premium features for this specific customer name\": {\n                    \"when_match\": true,\n                    \"conditions\": [\n                        {\n                            \"action\": \"EQUALS\",\n                            \"key\": \"customer_name\",\n                            \"value\": \"RanTheBuilder\"\n                        }\n                    ]\n                }\n            }\n        },\n        \"ten_percent_off_campaign\": {\n            \"default\": true\n        }\n    },\n    \"countries\": [\n        \"ISRAEL\",\n        \"USA\"\n    ]\n}",
    "ContentType": "application/json"
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confversion"
   }
  },
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfzero24C81C5A": {
   "Type": "AWS::AppConfig::DeploymentStrategy",
   "Properties": {
    "DeploymentDurationInMinutes": 0,
    "Description": "zero minutes, zero bake, 100 growth all at once",
    "FinalBakeTimeInMinutes": 0,
    "GrowthFactor": 100,
    "Name": "zero",
    "ReplicateTo": "NONE"
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confzero"
   }
  },
  "rootmainOrdersdynamicconfrootmainOrdersdynamicconfdeploy4DA73CE0": {
   "Type": "AWS::AppConfig::Deployment",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfOrders91C8FCAD"
    },
    "ConfigurationProfileId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfprofile6783FAA5"
    },
    "ConfigurationVersion": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfversion93674A9E"
    },
    "DeploymentStrategyId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfzero24C81C5A"
    },
    "EnvironmentId": {
     "Ref": "rootmainOrdersdynamicconfrootmainOrdersdynamicconfenvE9D41676"
    }
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/root-main-Ordersdynamic_conf/root-main-Ordersdynamic_confdeploy"
   }
  },
  "CDKMetadata": {
   "Type": "AWS::CDK::Metadata",
   "Properties": {
    "Analytics": "v2:deflate64:H4sIAAAAAAAA/1WMMQ+CMBBGfws7VIHJ0aCJI4HE1dRy4Em5a9qiIYT/LlUXpst77/Jl4pCJfSTfLlFNn2i8i7n2UvXxqm7SGMXUYifmoqWjMRqV9MgUr3imF1qmAcgHLL6Po/320nKLGoK/sPPQbOoVrPuPnMBonsJG7dcI3bS1S8AKHI9WwRKXk38w7XKRpiKPng4xsSN5HEBUv/sBQe2lP9EAAAA="
   },
   "Metadata": {
    "aws:cdk:path": "root-main-Orders/CDKMetadata/Default"
   }
  }
 },
 "Parameters": {
  "BootstrapVersion": {
   "Type": "AWS::SSM::Parameter::Value<String>",
   "Default": "/cdk-bootstrap/hnb659fds/version",
   "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
  }
 },
 "Rules": {
  "CheckBootstrapVersion": {
   "Assertions": [
    {
     "Assert": {
      "Fn::Not": [
       {
        "Fn::Contains": [
         [
          "1",
          "2",
          "3",
          "4",
          "5"
         ],
         {
          "Ref": "BootstrapVersion"
         }
        ]
       }
      ]
     },
     "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
    }
   ]
  }
 }
}

L2 version:

{
 "Resources": {
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfOrdersBEBC8B76": {
   "Type": "AWS::AppConfig::Application",
   "Properties": {
    "Name": "root-appconfig-Ordersdynamic_confOrders"
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confOrders/Resource"
   }
  },
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfenv2966CB14": {
   "Type": "AWS::AppConfig::Environment",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfOrdersBEBC8B76"
    },
    "Name": "dev"
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confenv/Resource"
   }
  },
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfzero6F77AA21": {
   "Type": "AWS::AppConfig::DeploymentStrategy",
   "Properties": {
    "DeploymentDurationInMinutes": 0,
    "FinalBakeTimeInMinutes": 0,
    "GrowthFactor": 100,
    "GrowthType": "LINEAR",
    "Name": "rootappconfigOrders-rootappcconfigOrdersdynamicconfzero-0E98E98A",
    "ReplicateTo": "NONE"
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confzero/Resource"
   }
  },
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversionConfigurationProfile9C631413": {
   "Type": "AWS::AppConfig::ConfigurationProfile",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfOrdersBEBC8B76"
    },
    "LocationUri": "hosted",
    "Name": "my_conf",
    "Type": "AWS.Freeform"
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confversion/ConfigurationProfile"
   }
  },
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversion4FD93E15": {
   "Type": "AWS::AppConfig::HostedConfigurationVersion",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfOrdersBEBC8B76"
    },
    "ConfigurationProfileId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversionConfigurationProfile9C631413"
    },
    "Content": "{\n    \"features\": {\n        \"premium_features\": {\n            \"default\": false,\n            \"rules\": {\n                \"enable premium features for this specific customer name\": {\n                    \"when_match\": true,\n                    \"conditions\": [\n                        {\n                            \"action\": \"EQUALS\",\n                            \"key\": \"customer_name\",\n                            \"value\": \"RanTheBuilder\"\n                        }\n                    ]\n                }\n            }\n        },\n        \"ten_percent_off_campaign\": {\n            \"default\": true\n        }\n    },\n    \"countries\": [\n        \"ISRAEL\",\n        \"USA\"\n    ]\n}",
    "ContentType": "application/json"
   },
   "UpdateReplacePolicy": "Retain",
   "DeletionPolicy": "Retain",
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confversion/Resource"
   }
  },
  "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversionDeployment9C0A7598163A3": {
   "Type": "AWS::AppConfig::Deployment",
   "Properties": {
    "ApplicationId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfOrdersBEBC8B76"
    },
    "ConfigurationProfileId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversionConfigurationProfile9C631413"
    },
    "ConfigurationVersion": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfversion4FD93E15"
    },
    "DeploymentStrategyId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfzero6F77AA21"
    },
    "EnvironmentId": {
     "Ref": "rootappconfigOrdersdynamicconfrootappconfigOrdersdynamicconfenv2966CB14"
    }
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/root-appconfig-Ordersdynamic_conf/root-appconfig-Ordersdynamic_confversion/Deployment9C0A7"
   }
  },
  "CDKMetadata": {
   "Type": "AWS::CDK::Metadata",
   "Properties": {
    "Analytics": "v2:deflate64:H4sIAAAAAAAA/22OsQrCMBiEn6V7Gmud3BQVHKUFV4npX/vbmIQkVUrIu2u0aIVOx91xx5fTZU59lrCHTXnVpgLP1JeO8Za8ohPTmitZ44X6TS3XWgvkzKGS5GV38o5GyRtIF+0WtFB9dKUzzMGlj+nmPe/Me3UwqkYBMd8r66D6a49g7HD9+wrRFmBVZzgEkqVM6IbRLFkNyLOoX86h9mPUMecE5ARJCOTQu0bJ2YLO53SRXC1iajrp8Aa0+OgTi/cdgTsBAAA="
   },
   "Metadata": {
    "aws:cdk:path": "root-appconfig-Orders/CDKMetadata/Default"
   }
  }
 },
 "Parameters": {
  "BootstrapVersion": {
   "Type": "AWS::SSM::Parameter::Value<String>",
   "Default": "/cdk-bootstrap/hnb659fds/version",
   "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
  }
 },
 "Rules": {
  "CheckBootstrapVersion": {
   "Assertions": [
    {
     "Assert": {
      "Fn::Not": [
       {
        "Fn::Contains": [
         [
          "1",
          "2",
          "3",
          "4",
          "5"
         ],
         {
          "Ref": "BootstrapVersion"
         }
        ]
       }
      ]
     },
     "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
    }
   ]
  }
 }
}

@peterwoodworth
Copy link
Contributor

Thanks, what catches my eye is that in the L2 version, the HostedVersionConfiguration has Update and Retention policies set to Retain. This would make sense that this is what causes it given the error.

this._cfnHostedConfigurationVersion.applyRemovalPolicy(RemovalPolicy.RETAIN);

You could remove this with escape hatches on the default child of the HostedConfiguration L2 construct as a workaround

@peterwoodworth peterwoodworth added p2 effort/small Small work item – less than a day of effort and removed p1 labels Aug 19, 2023
@ran-isenberg
Copy link
Author

Thanks, what catches my eye is that in the L2 version, the HostedVersionConfiguration has Update and Retention policies set to Retain. This would make sense that this is what causes it given the error.

this._cfnHostedConfigurationVersion.applyRemovalPolicy(RemovalPolicy.RETAIN);

You could remove this with escape hatches on the default child of the HostedConfiguration L2 construct as a workaround

Thanks @peterwoodworth that makes sense. I'd rather wait for the updated version. As I already have a working CFN, i dont want workarounds. I plan to write a blog post about the improved L2 version, and unless its flawless, i will not recommend it yet.

@chenjane-dev
Copy link
Contributor

Hi @ran-isenberg, thank you for bringing this to our attention. We are looking into it on our end.

@SteveJRice
Copy link

Thanks @ran-isenberg and @schmiddy . We are discussing internally how we want this work while still ensuring the best practices regarding resource removal. We will continue to let this bake in Alpha to get more feedback in general on these constructs

@ran-isenberg
Copy link
Author

Hey @SteveJRice , shouldn't the behavior be the same as the L1 constructs? Why do you require more feedback?
Imo, usually retain/delete policies are optional parameters with a default value.
In any case, resources are supposed to get removed at some point. The current state is not usable.

@ischia-exigenter
Copy link

Application, DeploymentStrategy, and Environment all have the usual applyRemovalPolicy method available. I would like to see this method added to the HostedConfiguration construct. We're using an escape hatch for now. Thanks

@ran-isenberg
Copy link
Author

I created a workaround gist that works for now if anyone cares:

gist

@amine-mf
Copy link
Contributor

@schmiddy and @ischia-exigenter, it is important to thumbs up the initial comment/message to show interest in the issue being fixed. (Prio is managed using those reactions)

@amine-mf
Copy link
Contributor

amine-mf commented Oct 16, 2023

I created a workaround gist that works for now if anyone cares:

gist

Thanks for the workaround, just a heads up, versions are "deleted" meaning, you will always keep only the latest version.

@chenjane-dev
Copy link
Contributor

Hi @ran-isenberg, this issue should be resolved now. Please confirm this is now working on your end

@ran-isenberg
Copy link
Author

@chenjane-dev what CDK version fixed it? i dont see anything in the 118/119 release notes about AppConfig

@chenjane-dev
Copy link
Contributor

@ran-isenberg your current version should be fine. We ended up releasing a fix server side

@ran-isenberg
Copy link
Author

@chenjane-dev yep, seems to be working!!!

Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-appconfig Related to AWS AppConfig bug This issue is a bug. effort/small Small work item – less than a day of effort p2
Projects
None yet
Development

No branches or pull requests

9 participants