From cab8b05bab2bf69380851cbf0b17e4f140975fb0 Mon Sep 17 00:00:00 2001 From: Nick Lynch Date: Wed, 1 Sep 2021 17:45:12 +0100 Subject: [PATCH] RFC 249: Rewrite the V2 Experiments RFC to have a more working-backwards tone (#377) While working on the release of the CDK v2 Experimental Modules, I found myself looking for a place to document and discuss some of the customer-facing decisions being made (e.g., package naming, versioning, Changelog organization). Unfortunately, the current state of RFC 249 does not lend itself well to this format (IMO); the RFC as-is is more implementation-focused and more heavily leans towards the why of many of the decisions. While absolutely useful context, it doesn't read as much as a working-backwards document as I'd like. My mental model to solve this was to copy (and save!) this really useful decision-making document, and replace the main RFC body with a series of working-backwards artifacts that (hopefully?) capture the customer experience of working with the alpha/unstable modules. --- text/0249-v2-experiments.expired.md | 1203 ++++++++++++++++++++++++ text/0249-v2-experiments.md | 1322 +++++---------------------- 2 files changed, 1448 insertions(+), 1077 deletions(-) create mode 100644 text/0249-v2-experiments.expired.md diff --git a/text/0249-v2-experiments.expired.md b/text/0249-v2-experiments.expired.md new file mode 100644 index 000000000..708894852 --- /dev/null +++ b/text/0249-v2-experiments.expired.md @@ -0,0 +1,1203 @@ +--- +feature name: Experimental APIs Post 1.x +start date: 2020-09-08 +rfc pr: https://github.com/aws/aws-cdk-rfcs/pull/250 +--- + +# Summary + +**⚠️ NOTE:** This RFC is expired and has been superceded by +. +This document is retained for historical context. + +--- + +When CDK version `2.0` is released to General Availability (GA), the single +monolithic Construct Library package we vend will no longer allow breaking +changes in its main modules. The purpose of this RFC is to discuss the +motivation behind this change, and to describe how API experiments will be +carried out in the CDK post `2.0`. + +# Motivation + +CDK releases contain a combination of stable and unstable features, which has +proven to be a pain point for customers. The AWS CDK packages are released +frequently -- at least once per week, sometimes more -- and each release +increments the minor version number (e.g. `1.59.0` to `1.60.0`). In the planned +`2.0` release of CDK, the main focus of the major version upgrade is to stop +packaging modules separately and to include them all in one package called +`aws-cdk-lib`. This will solve a number of problems related to peer dependencies +that make it harder to vend third-party libraries based on the CDK, but it does +not address the backwards compatibility problem caused by minor releases +containing breaking changes to unstable APIs. + +The CDK uses an exception to semantic versioning by labeling certain APIs (and +entire modules) as unstable, to allow us to make breaking changes to those APIs +in minor version upgrades. There is precedent for this in other open source +projects, but for CDK users, it has been a constant source of pain and +frustration. Users who do not carefully read and understand the documentation +simply install packages, copy sample code, make a few tweaks and put the code +into production. When they later upgrade to a new version, they are surprised to +find that their code no longer works. The perception of instability has driven +some users away, or caused them to limit their usage of CDK to the fundamental +L1 constructs, which do not provide them with the benefits of higher-level +abstractions. + +This RFC proposes that we stop releasing breaking changes in the main package we +vend. A user that installs `aws-cdk-lib` using NPM or `pip` or any other package +manager should be confident there will be no breaking changes in the `2.x` line +of releases for its lifetime. + +# Goals + +These are the goals of this RFC, in order from most to least important: + +## 1. Using CDK APIs that don't guarantee backwards-compatibility should require clear, explicit opt-in + +It should be absolutely obvious to a CDK customer when they are opting in to +using an API that might have backwards-incompatible changes in the future. From +experience, we have determined that including that information in the `ReadMe` +file of a module, or in the inline code documentation available in an +editor/IDE, does not meet the criteria of "absolutely obvious". + +If a customer is not aware of the stable vs unstable distinction, that means +they're using _only_ stable APIs, and that they will not be broken with minor +version CDK releases. + +## 2. We want to foster a vibrant ecosystem of third-party CDK packages + +In our estimation, the CDK cannot be successful without growing an expansive +collection of third-party packages that provide reusable Constructs on various +levels of abstraction. Changing to vending the Construct Library as a monolithic +package is one part of making that possible; we should make sure our approach to +unstable code also takes this into account. + +## 3. The CDK team can still perform API experiments + +We believe that one of the reasons for CDK's success is the ability to release +functionality quickly into the hands of customers, to get their feedback. The +ability to release experiments is crucial for that speed; if every single +released API decision carried with it the absolute burden of being 100% +backwards compatible, that would slow the pace of CDK innovation considerably +(especially third-party contributions), and would lengthen the feedback loop +from our customers on the quality of the proposed APIs. + +For those reasons, we consider it essential for the CDK team to retain the +capability to perform experiments with our APIs (of course, only those that are +clearly marked as such). + +## 4. Using experimental modules should be easy + +Our development methodology is highly dependent on feedback from the community +before finalizing APIs. To encourage users to use and provide feedback on +experimental APIs, we should make them easy to use. + +# Proposed changes + +To achieve the goals of this RFC, we propose the following changes: + +## 1. No more breaking changes in the main mono-CDK modules + +Because of a combination of how `aws-cdk-lib` will be depended on by third-party +libraries (through peer dependencies), and the goals of this RFC, it will no +longer be possible to make breaking changes to code inside `aws-cdk-lib`'s main +modules. + +(See Appendix A below for a detailed explanation why that is) + +### Option 6: API Previews + +AWS CDK v2 release notes: + +> Starting with version 2.0.0 of the AWS CDK, all modules and members will +> become stable. This means that from this release, we are committed to never +> introduce breaking changes in a non-major bump. +> +> One of the most common feedback we hear from customers is that they love how +> fast new features are added to the AWS CDK, we love it to. In v1, the +> mechanism that allowed us to add new features quickly was marking them as +> "experimental". Experimental features were not subject to semantic versioning, +> and we allowed breaking changes to be introduced in these APIs. This is the +> other most common feedback we hear from customers - breaking changes are not +> ok. +> +> #### Introducing API Previews +> +> To make sure we can keep adding features fast, while keeping our commitment to +> not release breaking changes, we are introducing a new model - API Previews. +> APIs that we want to get in front of developers early, and are not yet +> finalized, will be added to the AWS CDK with a specific suffix: `PreX`. APIs +> with the preview suffix will never be removed, instead they will be deprecated +> and replaced by either the stable version (without the suffix), or by a newer +> preview version. For example, assume we add the method +> `grantAwesomePowerPre1`: +> +> ```ts +> /** +> * This methods grants awesome powers +> */ +> grantAwesomePowerPre1(); +> ``` +> +> Times goes by, we get feedback that this method will actually be much better +> if it accept a `Principal`. Since adding a required property is a breaking +> change, we will add `grantAwesomePowerPre2()` and deprecate +> `grantAwesomePowerPre1`: +> +> ```ts +> /** +> * This methods grants awesome powers to the given principal +> * +> * @param grantee The principal to grant powers to +> */ +> grantAwesomePowerPre2(grantee: iam.IGrantable) +> +> /** +> * This methods grants awesome powers +> * @deprecated use grantAwesomePowerPre2 +> */ +> grantAwesomePowerPre1() +> ``` +> +> When we decide its time to graduate the API, the latest preview version will +> be deprecated and the final version - `grantAwesomePower` will be added. +> +> #### Alpha modules +> +> Writing the perfect API is hard, some APIs will require many iterations of +> breaking changes before they can be finalized, others may need a long bake +> time, and some both. This is especially true when writing a new L2. To make +> sure we don't make it too hard to contribute new L2s (keep em coming!), we are +> announcing the `AWS CDK Alpha modules` repo. This repo will be the home of AWS +> CDK modules which are in a very early stage of development. The alpha repo +> will contain multiple modules, each with its own prerelease version. When an +> alpha module is ready for prime time it will be added to `aws-cdk-lib`. Check +> out our contribution guide for more details. If you are writing a new L2, and +> you are not sure where it should be added, open a GitHub issue in the AWS CDK +> main repo. + +### Modules lifecycle + +This section discuss adding a new modules to `aws-cdk-lib`. Alpha and beta are +optional, modules can be added directly to `aws-cdk-lib` as stable. + +#### Alpha (optional) + +This stage is intended for modules that are in early development stage, they +change rapidly and may incurs many breaking changes. These modules will be +released separately from `aws-cdk-lib` and will have their own version, which +allows for breaking changes in accordance to semver, e.g `0.x` or prerelease +qualifiers. + +In order to release a module as alpha, the following conditions must be met: + +1. Module from `aws-cdk-lib` **can not** depend on it. +2. Can not depend on any other alpha module **\*\***. + +The purpose of the first condition is to prevent cyclic dependencies. To +illustrate the purpose of the second condition lets look at an example. Assume +we have an alpha module A, which depends on an alpha module B, graduating module +A means it will be added to `aws-cdk-lib`, since module A depends on module B, +it means that adding module A to `aws-cdk-lib` breaks condition 1 and creates a +cyclic dependency. To prevent the cycle we will have to graduate B as well, +which is not necessarily what we want. Additionally, if module B is an alpha +module, module A will have to declare a fixed dependency on it, locking its +consumers to the same version of module B - defeating the purpose of +peerDependencies. + +Condition 2 is marked with **\*\*** to note that there might be cases in which +it will make sense for unstable modules to depend on other unstable modules. +This is also evident from he fact that are several such dependencies in v1, see +[appendixes](#unstable-modules-depending-on-other-unstable-modules) for a full +list. + +Such cases will be considered on a per case basis. + +#### Beta (optional) + +In this stage modules are added to `aws-cdk-lib` with a preview suffix in the +name of the module, e.g `aws-batch-beta-x`. Modules in this stage are not +allowed to introduce any breaking changes to their API. Any non backward +compatible change will be introduced via deprecation\*\*. + +> \*\* _The deprecation process will be discussed in the API previews +> specification_ + +#### Stable + +In this stage modules are added to `aws-cdk-lib` under their final name, e.g +`aws-batch`. Addition of new APIs should follow the API previews specification. + +#### Multiple alpha modules VS. single module + +The question of whether we should release experimental modules under one package +or as separate modules has been discussed in this RFC. See +[Option 3](#option-3-separate-multiple-unstable-packages), and +[option 2](#option-2-separate-single-unstable-package). + +I suggest we release alpha modules as separate modules +([Option 3](#option-3-separate-multiple-unstable-packages)). The disadvantages +of this approach, listed in this RFC are: + +> 1. It's not possible for stable modules to depend on unstable ones (see +> Appendix B for data on how necessary that is for the CDK currently), with +> the same implications as above. +> 2. It's not possible for unstable modules to depend on other unstable modules +> (see Appendix B for data on how necessary that is for the CDK currently), +> as doing that brings us back to the dependency hell that mono-CDK was +> designed to solve. +> 3. Graduating a module to stable will be a breaking change for customers. We +> can mitigate this downside by keeping the old unstable package around, but +> that leads to duplicated classes. + +1 and 3 applies to both option 3 and option 2. + +Releasing each alpha module separately has the advantage of allowing consumers +to choose when they want to upgrade a specific module. If we release all modules +under a single package, in which every release may include breaking changes to +any numbers of its modules, users are forced to accept the breaking changes in +all modules, even if they only want to upgrade a single module. For example, +assume the uber package includes `aws-appmesh` and `aws-synthetics`, and that +release `0.x` of the uber package includes breaking changes to both modules, a +customer who only wants the new feature added to `aws-synthetics` now must +accept the breaking changes to `aws-appmesh`, which might include changes to +both the code and their deployed infrastructure they are not ready to make, e.g +resource replacement. Having separate module means reduces the blast radius of +every update. + +As for 2, if an unstable module need to depend on another unstable module, it +will do so using +[peerDependencies](https://docs.npmjs.com/cli/v6/configuring-npm/package-json#peerdependencies). +The "dependency hell" of v1 was a result of using dependencies where we should +have been using peerDependencies. `dependencies` will lead to multiple copies of +a library, `peerDependencies` will not. This subject has been discussed in +length during the RFC review, you can read more in this +[comment](https://github.com/aws/aws-cdk-rfcs/pull/279#discussion_r553581183). + +#### Migrating experimental modules to V2 + +Before v2 release we will need to decide the lifecycle stage of every v1 +experimental module. We will review all modules and devise a migration strategy +in a separate doc + +### Discussion + +Advantages: + +- Preview APIs can be used without declaring a fixed version on `aws-cdk-lib`. +- Since old versions of an API will only be deprecated and not removed, if + needed, we will be able to push critical updates to these versions as well. + For example, if we discover that `grantWrite` grants overly permissive + permissions, and the same occur in all of its experimental (deprecated) + predecessors, we will be able to push the fix to them as well. This will allow + us to get the fix to more customers in case of a critical fix. +- Libraries will be able to use preview APIs without locking their consumers to + a specific version of `aws-cdk-lib` (or any other module we vend). +- Same solution across all languages. + +Disadvantages: + +- Graduating a module will require a lot of code changes from our users to + remove the prefix/suffix. +- User does not have a clear motivation to upgrade to a newer version. +- Low adoption. Users might be hesitant to use an API with a `PreX` in its name, + assuming it means "will break". +- Cluttering the code base. Although most IDEs will mark deprecated properties + with a ~~strikethrough~~, they will still be listed by autocomplete. +- Will force some pretty long and ugly names on our APIs. Many previews APIs + will result in a less aesthetic user code. +- A lot of deprecated code in aws-cdk-lib, possibly blowing up `aws-cdk-lib`. + This might not be a real concern as we can reuse a lot the code between + different version of an API. + +**How can we encourage users to upgrade to a newer version?** + +When a new minor version introduces a breaking change to an API, users have a +clear motivation to upgrade to the new version of the API - they must do so in +order to upgrade to a newer version of the AWS CDK. If the API is only +deprecated, users have no motivation to upgrade to a newer version of the API, +even worse, they might not be aware that a newer version exists. One way to +encourage users to upgrade to a newer version of an API, is to supply tools that +will inform users. For example, we can add a capability to the `cdk doctor` +command that will notify users that their CDK application is using deprecated +experimental APIs. + +Executing `cdk doctor` will print: + +```bash +neta@dev/my-cdk-application$ cdk doctor + +Newer versions of APIs previews your application is using are available. You should consider upgrading. +To see which previews APIs can be upgraded, execute `cdk --show-deprecated-api-usage` +``` + +**If there are no breaking changes, are these APIs really experimental?** + +Given that preview APIs are safe to use, and no breaking changes will be +introduced to them, we might consider not referring to them in any special way, +and if needed, simply add a version to the API name. The first version of +`grantWrite` will be named `grantWrite`, the second version will be named +`grantWriteV1` and so on. While a preview API will not break, we should still +use a naming scheme that convey its non-final nature for the following reasons: + +1. **Real-estate**: `grantWrite` is a much better name than `grantWritePre3` for + the final API. +2. **Encourage feedback**: Declaring an API as "in preview" encourage the + community to supply feedback. +3. **Setting the right expectations**: When an API in preview users are aware + that this is not the final version of the API, and its deprecation is + expected. + +## 3. Extra unstable precautions + +This chapter discusses additional precautions we can choose to implement to +re-inforce goal #1 above. These are orthogonal to the decision on how to divide +the stable and unstable modules (meaning, we could implement any of these with +each of the options above). + +These could be added to either `@experimental` APIs in stable modules, to all +APIs in unstable modules, or both. + +### Require a feature flag for unstable code + +In this variant, we would add a runtime check into all unstable APIs that +immediately fails with an exception if the following context is missing: + +```json +{ + "context": { + "@aws-cdk:allowExperimentalFeatures": true + } +} +``` + +Note that `cdk init` will create a project with this context value set to +`false`. + +To avoid the manual and error-prone process of adding this check to every single +unstable API, we will need to modify JSII so that it recognizes the +`@experimental` decorator, and adds this check during compilation. + +Advantages: + +- Changing the context flag will be an explicit opt in from the customer to + agree to use unstable APIs. + +Disadvantages: + +- This will force setting the flag also for transitive experimental code (for + example, when an unstable API is used as an implementation detail of a + construct, but not in its public interface), which might be confusing. +- Since there is a single flag for all unstable code, setting it once might hide + other instances of using unstable code, working against stated goal #1. +- Requires changes in JSII. + +### Force a naming convention for unstable code + +We can modify `awslint` to force a certain naming convention for unstable code, +for example to add a specific prefix or suffix to all unstable APIs. + +Advantages: + +- Should fulfill goal #1 - it will be impossible to use an unstable API by + accident. +- Does not require changes in JSII, only in `awslint`. + +Disadvantages: + +- Will force some pretty long and ugly names on our APIs. +- Graduating a module will require a lot of code changes from our customers to + remove the prefix/suffix. + +# Appendix A - why can't we break backwards compatibility in the code of mono-CDK main modules? + +This section explains why it will not be possible to break backwards +compatibility of any API inside the stable modules of mono-CDK. + +Imagine we could break backwards compatibility in the code of the `aws-cdk-lib` +main modules. The following scenario would then be possible: + +Let's say we have a third-party library, `my-library`, that vends `MyConstruct`. +It's considered stable by its author. However, inside the implementation of +`MyConstruct`, it uses an experimental construct, `SomeExperiment`, from +mono-CDK's S3 module. It's just an implementation detail, though; it's not +reflected in the API of `MyConstruct`. + +`my-library` is released in version `2.0.0`, and it has a peer dependency on +`aws-cdk-lib` version `2.10.0` (with a caret, so `"aws-cdk-lib": "^2.10.0"`). + +Some time passes, enough that `aws-cdk-lib` is now in version `2.20.0`. A CDK +customer wants to use `my-library` together with the newest and shiniest +`aws-cdk-lib`,`2.20.0`, as they need some recently released features. However, +incidentally, in version `2.15.0` of `aws-cdk-lib`, `SomeExperiment` was broken +-- which is fine, it's an experimental API. Suddenly, the combination of +`my-library` `2.0.0` and `aws-cdk-lib` `2.20.0` will fail for the customer at +runtime, and there's basically no way for them to unblock themselves other than +pinning to version `2.14.0` of `aws-cdk-lib`, which was exactly the problem +mono-CDK was designed to prevent in the first place. + +# Appendix B - modules depending on unstable modules + +This section contains the snapshot of the interesting dependencies between +Construct Library modules as of writing this document. + +## Stable modules depending on unstable modules + +``` +⚠️ Stable module '@aws-cdk/aws-applicationautoscaling' depends on unstable module '@aws-cdk/aws-autoscaling-common' +⚠️ Stable module '@aws-cdk/aws-autoscaling' depends on unstable module '@aws-cdk/aws-autoscaling-common' +⚠️ Stable module '@aws-cdk/aws-events-targets' depends on unstable module '@aws-cdk/aws-batch' +⚠️ Stable module '@aws-cdk/aws-lambda' depends on unstable module '@aws-cdk/aws-efs' +⚠️ Stable module '@aws-cdk/aws-stepfunctions-tasks' depends on unstable module '@aws-cdk/aws-batch' +⚠️ Stable module '@aws-cdk/aws-stepfunctions-tasks' depends on unstable module '@aws-cdk/aws-glue' +``` + +## Unstable modules depending on other unstable modules + +``` +ℹ️️ Unstable package '@aws-cdk/aws-apigatewayv2-integrations' depends on unstable package '@aws-cdk/aws-apigatewayv2' +ℹ️️ Unstable package '@aws-cdk/aws-appmesh' depends on unstable package '@aws-cdk/aws-acmpca' +ℹ️️ Unstable package '@aws-cdk/aws-backup' depends on unstable package '@aws-cdk/aws-efs' +ℹ️️ Unstable package '@aws-cdk/aws-docdb' depends on unstable package '@aws-cdk/aws-efs' +ℹ️️ Unstable package '@aws-cdk/aws-ses-actions' depends on unstable package '@aws-cdk/aws-ses' +``` + +# Appendix C - discarded solutions to problem #2 + +These potential solutions to problem #2 were discarded by the team as not +viable. + +## Option 1: separate submodules of `aws-cdk-lib` + +In this option, we would use the namespacing features of each language to vend a +separate namespace for the experimental APIs. The customer would have to +explicitly opt-in by using a language-level import of a namespace with +"experimental" in the name. + +Example using stable and unstable Cognito APIs: + +```ts +import * as cognito from 'aws-cdk-lib/aws-cognito'; +import * as cognito_preview from 'aws-cdk-lib/experimental/aws-cognito'; + +const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); +const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; +const userPoolClient = new cognito.UserPoolClient(...); +``` + +Advantages: + +1. It's possible for stable module to depend on unstable ones (see Appendix B + for data on how necessary that is for the CDK currently) +2. It's possible for unstable modules to depend on other unstable modules (see + Appendix B for data on how necessary that is for the CDK currently). + +Disadvantages: + +1. Might be considered less explicit, as a customer never says they want to + depend on a package containing unstable APIs, or with `0.x` for the version. +2. If a third-party package depends on an unstable API in a non-obvious way (for + example, only in the implementation of a construct, not in its public API), + that might break for customers when upgrading to a version of `aws-cdk-lib` + that has broken that functionality compared to the `aws-cdk-lib` version the + third-party construct is built against (basically, the same scenario from + above that explains why we can no longer have unstable code in stable + mono-CDK modules). None of the options solve the problem of allowing + third-party libraries to safely depend on unstable Construct Library code; + however, the fact that all unstable code in this variant is shipped in + `aws-cdk-lib` makes this particular problem more likely to manifest itself. +3. Graduating a module to stable will be a breaking change for customers. We can + mitigate this downside by keeping the old unstable module around, but that + leads to duplicated classes in the same package. + +**Verdict**: discarded because disadvantage #2 was considered a show-stopper. + +## Option 4: separate V3 that's all unstable + +In this option, we will fork the CDK codebase and maintain 2 long-lived +branches: one for version `2.x`, which will be all stable, and one for version +`3.x`, which will be all unstable. + +Example using stable and unstable Cognito APIs: (assuming the dependency on +`"aws-cdk-lib"` is in version `"3.x.y"`): + +```ts +import * as cognito from 'aws-cdk-lib/aws-cognito'; + +const idp = new cognito.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); +const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; +const userPoolClient = new cognito.UserPoolClient(...); +``` + +Advantages: + +1. It's possible for unstable modules to depend on other unstable modules (see + Appendix B for data on how necessary that is for the CDK currently). + +Disadvantages: + +1. It's not possible for stable modules to depend on unstable ones (with the + same implications as above). +2. Does not make it obvious to customers that this is unstable (`3.x` is + considered stable in semantic versioning). +3. We are going from "some code is stable, some is unstable" to "all of this is + unstable", which seems to be against the customer feedback we're hearing + that's the motivation for this RFC. +4. Two long-lived Git branches will mean constant merge-hell between the two, + and since `3.x` has free rein to change anything, there will be a guarantee + of constant conflicts between the two. +5. Fragments the mono-CDK third-party library community into two. +6. Very confusing when we want to release the next major version of the CDK (I + guess we go straight to `4.x`...?). +7. The fact that all code in `3.x` is unstable means peer dependencies don't + work (see above for why). +8. Graduating a module to stable will be a breaking change for customers. We can + mitigate this downside by keeping the old unstable package around, but that + leads to duplicated classes between the 2 versions. + +**Verdict**: discarded as a worse version of option #5. + +### Option 2: separate single unstable package + +Instead of vending the unstable modules together with the stable ones, we can +vend a second mono-CDK, `aws-cdk-lib-experiments` (actual name can be changed +before release of course). A customer will have to explicitly depend on +`aws-cdk-lib-experiments`, which will be released in version `0.x` to make it +even more obvious that this is unstable code. `aws-cdk-lib-experiments` would +have a caret peer dependency on `aws-cdk-lib`. + +Example using stable and unstable Cognito APIs: + +```ts +import * as cognito from 'aws-cdk-lib/aws-cognito'; +import * as cognito_preview from 'aws-cdk-lib-experiments/aws-cognito'; + +const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); +const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; +const userPoolClient = new cognito.UserPoolClient(...); +``` + +Advantages: + +1. Very explicit (customer has to add a dependency on a package with + "experiments" in the name and version `0.x`). +2. It's possible for unstable modules to depend on other unstable modules (see + Appendix B for data on how necessary that is for the CDK currently). + +Disadvantages: + +1. It's not possible for stable modules to depend on unstable ones (see Appendix + B for data on how necessary that is for the CDK currently). This has serious + side implications: + + - All unstable modules that have stable dependents today will have to be + graduated before `v2.0` is released. + - Before a module is graduated, all of its dependencies need to be graduated. + - It will not be possible to add new dependencies on unstable modules to + stable modules in the future (for example, that's a common need for + StepFunction Tasks). + +2. Graduating a module to stable will be a breaking change for customers. We can + mitigate this downside by keeping the old unstable module around, but that + leads to duplicated classes. + +### Option 3: separate multiple unstable packages + +In this option, each experimental library will be vended as a separate package. +Each would have the name "experiments" in it (possible naming convention: +`@aws-cdk-lib-experiments/aws-`), and would be released in version +`0.x` to make it absolutely obvious this is unstable code. Each package would +declare a caret peer dependency on `aws-cdk-lib`. + +Example using stable and unstable Cognito APIs: + +```ts +import * as cognito from 'aws-cdk-lib/aws-cognito'; +import * as cognito_preview from '@aws-cdk-lib-experiments/aws-cognito'; + +const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); +const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; +const userPoolClient = new cognito.UserPoolClient(...); +``` + +Advantages: + +1. Very explicit (customer has to add a dependency on a package with + "experiments" in the name and version `0.x`). +2. This is closest to the third-party CDK package experience our customers will + have. + +Disadvantages: + +1. It's not possible for stable modules to depend on unstable ones (see Appendix + B for data on how necessary that is for the CDK currently), with the same + implications as above. +2. It's not possible for unstable modules to depend on other unstable modules + (see Appendix B for data on how necessary that is for the CDK currently), as + doing that brings us back to the dependency hell that mono-CDK was designed + to solve. +3. Graduating a module to stable will be a breaking change for customers. We can + mitigate this downside by keeping the old unstable package around, but that + leads to duplicated classes. + +### Option 5: superset/subset releases + +Instead of _splitting_ the stable and experimental APIs into two different +packages, we release two builds of the same source code, where the +_experimental_ build is a superset of the _stable_ build. + +This can be done automatically, by using jsii to strip out experimental APIs +from the **public interface** of the stable release. The implementation (`.js` +files) will be the same in both releases, so stable APIs in `aws-cdk-lib` can +safely use experimental APIs internally (though not in their public API +surfaces). + +#### How does stripping APIs from the public work? + +| TypeScript | JavaScript | Java, Python, C#, Go | +| --------------------- | ---------- | ------------------------------- | +| Stripped from `.d.ts` | N/A | No bindings generated by `jsii` | + +In TypeScript, the compiler will prevent usage of APIs that are not advertised +in the `.d.ts` files. Users will be able to bypass this guarantee by using +`as any` though. + +For JavaScript, an IDE will (hopefully¹) not autocomplete APIs missing from the +`.d.ts` files; otherwise nothing prevents users from calling them. On the other +hand, JavaScript devs are by definition required to read the docs, so they can't +miss the _experimental_ banners displayed there. + +For jsii client languages, the bindings for experimental APIs will simply not +exist so there's no way to work around that (though we'll have to check whether +for example Python makes it possible to pass struct values that aren't in the +declared API surface). + +¹) A _smart enough_ IDE could discover those APIs by parsing the JavaScript +directly, or by doing dynamic execution and runtime inspection of objects. This +was already being done before TypeScript existed, I did not survey the current +landscape of IDEs to figure out which ones use what techniques to provide +autocomplete for users. + +#### What about experimental struct properties + +We also commonly use `@experimental` to indicate experimental struct properties +for `Props` types. + +There will be code that users can write that will pass a props object that +technically contains non-advertised, experimental properties. Without additional +runtime support the object constructor will happily interpret the presence of +those properties and make decisions based off of them. + +An example in TypeScript: + +```ts +// Construct +interface StableConstructProps { + readonly stableProp: string; + + /** @experimental */ + readonly experimentalProp: string; +} + +class StableConstruct { + constructor(scope: Construct, id: string, props: StableConstructProps) { + super(scope, id); + if (props.experimentalProp) { + console.log('Experimental features activated!'); + } + } +} + +//---------------------------------------------------------------------- +// THE FOLLOWING CLIENT CODE IS USING THE STABLE BUILD +// +// As far as this client code is aware, StableConstructProps looks like this: +// +// interface StableConstructProps { +// readonly stableProp: string; +// } + +const props = { + stableProp: 'foo', + experimentalProp: 'bar', +}; + +// Even though we're using "stable mode", the following code still +// prints 'Experimental features activated!'. +// +// The call below is allowed because 'typeof props' is a superset of StableConstructProps. +new StableConstruct(this, 'Id', props); + +// In contrast the call below would NOT be allowed: +new StableConstruct(this, 'Id', { + stableProp: 'foo', + experimentalProp: 'bar', +}); +``` + +We would need runtime support +(`if (props.experimentalProp && IN_EXPERIMENTAL_MODE) { ... }`) to detect and +prevent this, or accept it. + +The example above is for TypeScript; it obviously also holds for JavaScript, and +_might_ hold for jsii languages in certain cases. + +#### Vending + +There are a number of different ways we can choose to distinguish the two +builds. See Appendix D for an evaluation of all the possibilities. + +- Distinguish by package name + - Can't be used in **pip**: there's no way to have an app that uses the + experimental library work with a library that uses the stable library (no + way to prevent the transitive dependency package with a different name from + being installed). +- Distinguish by prerelease tag + - **NuGet**: requires that experimental version semver-sorts _after_ the + matching stable version, otherwise it errors out and fails the restore + operation. + - **npm**: will always complain when trying to satisfy a stable requirement + with an experimental version, regardless of how it sorts. + - **Maven**: does not recognize prerelease tags. This means that _apps_ can't + have an open-ended version range since they might accidentally pick up + experimental versions (a Maven range of `[1.60.0, 2.0.0)` _will_ match + `1.61.0-experimental`). This might not be a problem as this is rarely done + in practice (since Maven doesn't have a lock file, the `pom` itself serves + as the lock file). +- Distinguish by major version + - Very similar to prerelease tags, except we don't depend on Package Manager's + support for prerelease tags; all PMs support major versions. + +Even though there are downsides, the prerelease tag is the least bad of the +alternatives and the one that's (probably) easiest to explain. + +We have to work around the version ordering problem though, by making sure the +experimental version sorts after the stable version. We can either use the minor +or the patch version: + +```shell +# Experimental uses minor bump +1.60.0 < 1.61.0-experimental + +# Experimental uses (fake) patch bump +1.60.0 < 1.60.100-experimental +``` + +#### Summary + +Advantages: + +1. It's possible for stable modules to depend on unstable ones (see Appendix B + for data on how necessary that is for the CDK currently). +2. Stabilizing code will not be a breaking change for customers. +3. Stabilizing code will be a simple operation for CDK developers (remove an + annotation). +4. No additional management overhead of multiple packages. +5. It's possible for unstable modules to depend on other unstable modules. + +Disadvantages: + +1. Stable/experimental versioning scheme is not based on any well-known industry + standard, we're going to have to clearly explain it to people. +2. Requires changes to jsii +3. Requires changes to the docs to make switching between editions possible +4. Protection not offered for JavaScript users, can be bypassed in TypeScript; + although something is to be said for it being the same with `private` fields + today. +5. Does not satisfy + [goal 4 - Using experimental modules should be easy](#4-using-experimental-modules-should-be-easy). + Customers that wants to use a single experimental API must pay the cost of + using a different version of the **entire** `aws-cdk-lib`. This is a non + trivial cost due to the following: + - Migrating to either a new major version, or a prerelease version, is a + process that usually includes accepting lot of breaking changes. While this + may not be the case for `aws-cdk-lib`, users wil be still hesitant as it is + the standard meaning of such migration. + - Users which uses the experimental version might use experimental APIs they + didn't intend to, similar to the experimental experience in v1. + +#### POC results + +> _Option 5 should be move to the appendices section, it is added here for the +> purpose of minimizing the diff in the review process. It will be moved once +> the RFC is finalized._ + +Option 5 was rejected since there is no way to model the relationship between +the "experimental" version and the "stable" version through semantic versioning. + +##### Prerelease qualifiers + +According to the semver [specification](https://semver.org/#spec-item-11), a +precedence between versions is calculated **only** if the **major**, **minor** +and **patch** are equal. This means that there is no way for a constructs +library to declare a range of supported versions which include an experimental +version. Sticking to the above example, a CDK construct library declaring a peer +dependency on version `^1.60.0` of `aws-cdk-lib`, can not be used in a CDK +application that declares a dependency on `aws-cdk-lib` version +`1.61.0-experimental`. This is because the patch part in `1.60.0` and +`1.61.0-experimental` is not equal, which means `1.61.0-experimental` does not +satisfy `^1.60.0`. In npm versions prior to npm-v7, if `peerDependencies` +requirements are not met, executing `npm install` would only issue a warning. In +npm-v7, which automatically tries to install `peerDependencies`, executing +`npm install` will throw an error. + +To illustrate the user experience with npm-v7. The below is the output of +executing `npm install` in a CDK application (`my-cdk-application`), which +declares a dependency on a `aws-cdk-lib` version `1.61.0-experimental`, and on a +CDK library (`my-construct-lib`) which itself declares a peer dependency on +`aws-cdk-lib` version `^1.60.0`: + +``` +npm ERR! code ERESOLVE +npm ERR! ERESOLVE unable to resolve dependency tree +npm ERR! +npm ERR! While resolving: my-cdk-application@1.0.0 +npm ERR! Found: aws-cdk-lib@1.61.0-experimental +npm ERR! node_modules/aws-cdk-lib +npm ERR! aws-cdk-lib"@1.61.0-experimental" from the root project +npm ERR! +npm ERR! Could not resolve dependency: +npm ERR! peer aws-cdk-li@"^1.60.0" from my-construct-lib@1.6.0 +npm ERR! node_modules/my-construct-lib +npm ERR! my-construct-lib@"1.6.0" from the root project +npm ERR! +npm ERR! Fix the upstream dependency conflict, or retry +npm ERR! this command with --force, or --legacy-peer-deps +npm ERR! to accept an incorrect (and potentially broken) dependency resolution. +npm ERR! +``` + +Users can work around this by executing `npm install --force` which means that +npm will not check for version compatibility at all, and therefore not an +acceptable solution. + +##### Separate major versions + +This was rejected due to similar reasons as listed in the disadvantages of +[Option 4: separate V3 that's all unstable](#option-4-separate-v3-thats-all-unstable). + +# Appendix D - Alternatives for identifying stable/experimental builds + +We have the following options to pick from: + +- Different package names (`aws-cdk-lib` vs `aws-cdk-lib-experimental`). +- Pre-release version tag +- Different major versions + +We need to evaluate all of these, for each package manager, on the following +criteria: + +- Can applications depend on an upwards unbounded range of _stable_ builds? + (important) +- Can an application using the _experimental_ build use a library using the + _stable_ build? (important) + - Without the package manager complaining (preferably) +- Can a library declare a tool-checked dependency on a specific _experimental_ + version? (preferably) + +We have ruled out the requirement that an application using _stable_ may +transparently use a library using _experimental_. We will disallow this as it's +not feasible. We're trying to achieve the following compatibility matrix: + +| . | Stable App | Experimental App | +| ---------------- | ----------------- | ------------------------ | +| Stable Lib | floating versions | lib floating, app pinned | +| Experimental Lib | - | pinned versions | + +> NOTE: in the following sections, I'll be using the terms "stable app", +> "experimental app", "stable lib" and "experimental lib" as stand-ins for the +> more accurate but much longer "app that depends on stable CDK", "app that +> depends on experimental CDK", etc. + +## Different package names + +We would vend two different package names, for example: + +- `aws-cdk-lib` +- `aws-cdk-lib-experimental` + +The tl;dr of this section is as follows. Read below for details. + +| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | +| ----- | ----------- | -------------------------------- | --------------------- | --------------------------- | +| NPM | yes | with package aliases | yes | no | +| Maven | yes | with excluded dependencies | yes | yes | +| pip | yes | no | - | no | +| NuGet | yes | no | no | yes | +| Go | ? | ? | | ? | + +### NPM\* + +In NPM-based languages (JavaScript and TypeScript), the package name appears in +source code (in the `require()` statement). If we changed the library name, we +wouldn't be able to use a library that uses stable code (it would contain +`require('aws-cdk-lib')`) in an application that uses experimental code: + +```js +//------------- LIBRARY THAT USES STABLE --------------- +// package.json +{ + "peerDependencies": { + "aws-cdk-lib": "^1.60.0" + } +} + +// index.ts +import * as cdk from 'aws-cdk-lib'; // <- locally perfectly sane + +//------------- APPLICATION THAT USES EXPERIMENTAL AND THIS LIB --------------- +// package.json +{ + "devDependencies": { + "lib-that-uses-stable": "^1.2.3", + "aws-cdk-lib-experimental": "1.60.0" + } +} + +// app.ts +import * as cdk from 'aws-cdk-lib-experimental'; // <- locally perfectly sane +``` + +Well oops. Turns out we can't have a library that depends on `aws-cdk-lib` in an +application that uses `aws-cdk-lib-experimental`; NPM will complain about the +missing `peerDependency`, and the `require('aws-cdk-lib')` call will just fail +at runtime. + +NPM 6.9.0 (released May 2019) introduces a feature that helps with this: +[package aliases](https://github.com/npm/rfcs/blob/latest/implemented/0001-package-aliases.md) +which allows installing a package under a different name, so we could install +`aws-cdk-lib-experimental` under the name `aws-cdk-lib`. The application +`package.json` will now look like this: + +```js +//------------- APPLICATION THAT USES EXPERIMENTAL AND STABLE LIB --------------- +// package.json +{ + "devDependencies": { + "lib-that-uses-stable": "^1.2.3", + "aws-cdk-lib": "npm:aws-cdk-lib-experiments@1.60.0", + } +} + +// app.ts +import * as cdk from 'aws-cdk-lib'; +``` + +So that's cool. + +What would a construct library that depends on an experimental build look like, +and could we use that in the same application? + +An experimental library would declare this: + +```json +{ + "peerDependencies": { + "aws-cdk-lib": "1.60.0" + // Note: ideally you'd like to express "aws-cdk-lib-experiments": "1.60.0" here, but + // if you do that "npm ls" will complain about an unmet peer dependency + }, + "devDependencies": { + "aws-cdk-lib": "npm:aws-cdk-lib-experiments@1.60.0" + } +} +``` + +This will all work as intended and put code in the right places. + +However, we lose the tool-checked dependency on experimental: from NPM's point +of view the library depends on _stable_ CDK (whereas we want it to declare that +it depends on _experimental_ CDK). It's only notes in the README that can inform +the consumer otherwise. + +### Maven + +No need to change the sources when using a stable library with an experimental +app, as the namespaces and class names will be the same. + +Using a stable library in an experimental app can be done in two ways: + +- The library uses the `provided` dependency flavor (similar to + `peerDependencies` in NPM). Requires the app to bring the right version of + CDK. +- The app that wants to use the _experimental_ build uses dependency exclusions + to remove the dependency on the _stable_ build (looks like this is necessary + for every stable library individually) + +[Reference](https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html#) + +### pip + +No need to change the sources when using a stable library with an experimental +app, as the namespaces and class names will be the same. + +`pip` transitively fetches from the library's `setup.py`, and there's no way to +override that. + +Our only option around that is to have stable libraries not declare any CDK +dependency in `setup.py`, but then we also lose the ability for libraries to +specify their dependency version, which is not really acceptable. + +Effectively, this seems to be a no-go. + +### .NET Core/NuGet + +No need to change the sources when using a stable library with an experimental +app, as the namespaces and class names will be the same. + +...pending responses from .NET SDK team... + +### Go + +??? + +## Prerelease tags + +We would vend the experimental build under a prerelease tag, for example: + +- `1.60.0` +- `1.60.0-experimental` + +Because of semver ordering, if we use the numbering above then the +`experimental` version will sort _before_ the stable version, which may lead to +problems because a library's dependency requirement of `^1.60.0` would _not_ be +satisfied by a version numbered `1.60.0-experimental`. + +We could get rid of some of the ordering and warning problems by making sure the +experimental version semver-orders _after_ the stable version by making sure we +bump some number: + +- `aws-cdk-lib@1.60.100-experimental` +- `aws-cdk-lib@1.61.0-experimental` + +| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | +| ----- | ------------------ | -------------------------------- | --------------------------- | --------------------------- | +| NPM | yes | yes | no | yes | +| Maven | no but that's okay | yes | yes | yes | +| pip | yes | yes | if experimental sorts later | yes | +| NuGet | yes | if experimental sorts later | yes | yes | +| Go | ? | ? | ? | ? | + +### NPM + +Can apps use stable ranges: yes, `^1.60.0` will not auto-pick +`1.61.1-experimental` for downloading. + +Experimental app uses stable lib: can be done, the lib only uses a +`peerDependency`. + +However, if the `peerDependency` of a library is stable (`^1.60.0`) and we're +trying to satisfy it with an experimental version, `npm ls` will complain +_regardless_ of where the experimental version sorts with respect to the stable +version. + +It will _work_, but npm will _complain_. + +### Maven + +Stable ranges: no, `[1.60.0,)` _will_ include `1.61.1-experimental`. However, +this might not be an issue as `pom.xml` typically serves as the lock file and +people will not build applications with open-ended ranges in the pom file. + +If we _wanted_ to, we could vend as `1.60.0-experimental-SNAPSHOT`; snapshot +versions will not match a range unless specifically requested. However, +`SNAPSHOT` versions are not intended for this, they are intended to indicate +mutability and bypass caches: every configurable time period (by default, every +day but can be every build, and snapshots can also be disabled altogether) the +`SNAPSHOT` version will be fetchd again, as it is assumed to be mutable. + +Maven uses a "closest to root wins" dependency model, so the application can +substitute an experimental version in place of the declared stable +compatibility. + +It will not complain about incompatible versions unless you really _really_ ask +for diagnostics (and even then it's hard to get it to show an error). + +### pip + +Stable ranges: yes, `>=1.60.0` will not match version `1.60.1-experimental` +[ref](https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions) + +Experimental app uses stable lib: `pip` allows overriding transitive +dependencies using `requirements.txt`. + +It will complain if the experimental version sorts before the stable version, +but will NOT complain if the experimental version sorts after the stable +version. Everything still gets installed, even if it complains. + +Example of a complaint (using a different package): + +```text +awscli 1.18.158 has requirement s3transfer<0.4.0,>=0.3.0, but you'll have s3transfer 0.2.0 which is incompatible. +``` + +(Note: when trying the PyPI testing server, the version number schema I had in +mind: `X.Y.Z-experimental` is not accepted, nor is `X.Y-experimental` or +`X.Yexp`. Ultimately I had to go for `1.60rc1`.) + +### NuGet + +PJ Pittle from the .NET team has confirmed for us that NuGet will complain and +**error out** if we use: + +- Library depends on CDK `>= 1.60.0` +- App uses CDK `1.60.0-experimental` + +(Because of semver ordering.) + +The only solution seems to be to make sure that the experimental version sorts +after the stable version. + +## Different major versions + +We would vend the experimental build under a prerelease tag, for example: + +- `aws-cdk-lib@2.60.0` +- `aws-cdk-lib@3.60.0` (odd ranges are experimental) + +Or: + +- `aws-cdk-lib@2.60.0` +- `aws-cdk-lib@102.60.0` (100+ versions are experimental) + +| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | +| ----- | ----------- | -------------------------------- | --------------------- | --------------------------- | +| NPM | yes | yes | yes | yes | +| Maven | yes | yes | yes | yes | +| pip | yes | no | - | yes | +| NuGet | yes | yes | ? | yes | +| Go | ? | ? | ? | ? | + +### NPM + +Mostly like pre-release tags, except libraries using stable can explicitly +declare they're usable with both stable and experimental ranges by using +multiple `peerDependency` ranges: + +```json +{ + "peerDependencies": { + "aws-cdk-lib": "^2.60.0 ^102.60.0" + } +} +``` + +This would suppress the warning you would otherwise get for a non-compatible +version. + +### Maven + +Like pre-release tags, except we don't get the mixing in of pre-release versions +with regular versions. + +### pip + +Like pre-release tags. + +### NuGet + +Like pre-release tags. + +### Go + +? diff --git a/text/0249-v2-experiments.md b/text/0249-v2-experiments.md index 81a3a03dc..29ecd94a0 100644 --- a/text/0249-v2-experiments.md +++ b/text/0249-v2-experiments.md @@ -1,111 +1,37 @@ --- -feature name: Experimental APIs Post 1.x -start date: 2020-09-08 -rfc pr: https://github.com/aws/aws-cdk-rfcs/pull/250 +rfc pr: [#xxx](https://github.com/aws/aws-cdk-rfcs/pull/xxx) <-- fill this after you've already created the PR +tracking issue: https://github.com/aws/aws-cdk-rfcs/issues/249 --- -# Summary - -When CDK version `2.0` is released to General Availability (GA), the single -monolithic Construct Library package we vend will no longer allow breaking -changes in its main modules. The purpose of this RFC is to discuss the -motivation behind this change, and to describe how API experiments will be -carried out in the CDK post `2.0`. - -# Motivation - -CDK releases contain a combination of stable and unstable features, which has -proven to be a pain point for customers. The AWS CDK packages are released -frequently -- at least once per week, sometimes more -- and each release -increments the minor version number (e.g. `1.59.0` to `1.60.0`). In the planned -`2.0` release of CDK, the main focus of the major version upgrade is to stop -packaging modules separately and to include them all in one package called -`aws-cdk-lib`. This will solve a number of problems related to peer dependencies -that make it harder to vend third-party libraries based on the CDK, but it does -not address the backwards compatibility problem caused by minor releases -containing breaking changes to unstable APIs. - -The CDK uses an exception to semantic versioning by labeling certain APIs (and -entire modules) as unstable, to allow us to make breaking changes to those APIs -in minor version upgrades. There is precedent for this in other open source -projects, but for CDK users, it has been a constant source of pain and -frustration. Users who do not carefully read and understand the documentation -simply install packages, copy sample code, make a few tweaks and put the code -into production. When they later upgrade to a new version, they are surprised to -find that their code no longer works. The perception of instability has driven -some users away, or caused them to limit their usage of CDK to the fundamental -L1 constructs, which do not provide them with the benefits of higher-level -abstractions. - -This RFC proposes that we stop releasing breaking changes in the main package we -vend. A user that installs `aws-cdk-lib` using NPM or `pip` or any other package -manager should be confident there will be no breaking changes in the `2.x` line -of releases for its lifetime. - -# Goals +# CDKv2 Experiments -These are the goals of this RFC, in order from most to least important: - -## 1. Using CDK APIs that don't guarantee backwards-compatibility should require clear, explicit opt-in - -It should be absolutely obvious to a CDK customer when they are opting in to -using an API that might have backwards-incompatible changes in the future. From -experience, we have determined that including that information in the `ReadMe` -file of a module, or in the inline code documentation available in an -editor/IDE, does not meet the criteria of "absolutely obvious". - -If a customer is not aware of the stable vs unstable distinction, that means -they're using _only_ stable APIs, and that they will not be broken with minor -version CDK releases. - -## 2. We want to foster a vibrant ecosystem of third-party CDK packages - -In our estimation, the CDK cannot be successful without growing an expansive -collection of third-party packages that provide reusable Constructs on various -levels of abstraction. Changing to vending the Construct Library as a monolithic -package is one part of making that possible; we should make sure our approach to -unstable code also takes this into account. - -## 3. The CDK team can still perform API experiments - -We believe that one of the reasons for CDK's success is the ability to release -functionality quickly into the hands of customers, to get their feedback. The -ability to release experiments is crucial for that speed; if every single -released API decision carried with it the absolute burden of being 100% -backwards compatible, that would slow the pace of CDK innovation considerably -(especially third-party contributions), and would lengthen the feedback loop -from our customers on the quality of the proposed APIs. - -For those reasons, we consider it essential for the CDK team to retain the -capability to perform experiments with our APIs (of course, only those that are -clearly marked as such). - -## 4. Using experimental modules should be easy - -Our development methodology is highly dependent on feedback from the community -before finalizing APIs. To encourage users to use and provide feedback on -experimental APIs, we should make them easy to use. +As part of the CDK v2 release, the CDK is adopting a new method of creating, developing, and releasing new L2 +constructs. CDK v1 intermixed stable modules and constructs alongside experimental constructs, confusing customers and +breaking semver semantics. -# Proposed changes +For CDK v2, the main CDK artifact (`aws-cdk-lib`) will contain only stable, consistent APIs +and constructs that adhere to backwards compatibility. Customers can consume any APIs from `aws-cdk-lib` and have +confidence that no breaking changes will be introduced (without a major version bump). -To achieve the goals of this RFC, we propose the following changes: +All current and new CDK modules +being developed — and not yet stable — it will be released as its own separate artifact, and versioned appropriately +according to semver. -## 1. No more breaking changes in the main mono-CDK modules +In addition, this RFC introduces a new standard method for previewing new APIs within `aws-cdk-lib`. -Because of a combination of how `aws-cdk-lib` will be depended on by third-party -libraries (through peer dependencies), and the goals of this RFC, it will no -longer be possible to make breaking changes to code inside `aws-cdk-lib`'s main -modules. +## Working Backwards -(See Appendix A below for a detailed explanation why that is) +Given the breadth of this change, several Working Backwards artifacts are presented here, each targeting a different +aspect of the customer experience when consuming experimental APIs from either `aws-cdk-lib` or one of the new +alpha modules. -### Option 6: API Previews +### Release Notes -AWS CDK v2 release notes: +The following is a hypothetical snippet from the CDK v2 Release Notes: -> Starting with version 2.0.0 of the AWS CDK, all modules and members will -> become stable. This means that from this release, we are committed to never -> introduce breaking changes in a non-major bump. +> Starting with version 2.0.0 of the AWS CDK, all modules and members vended +> as part of the main CDK library (`aws-cdk-lib`) will always be stable; we are +> committing to never introduce breaking changes in a non-major bump. > > One of the most common feedback we hear from customers is that they love how > fast new features are added to the AWS CDK, we love it to. In v1, the @@ -115,28 +41,28 @@ AWS CDK v2 release notes: > other most common feedback we hear from customers - breaking changes are not > ok. > -> #### Introducing API Previews +> **Introducing API Previews** > > To make sure we can keep adding features fast, while keeping our commitment to > not release breaking changes, we are introducing a new model - API Previews. > APIs that we want to get in front of developers early, and are not yet -> finalized, will be added to the AWS CDK with a specific suffix: `PreX`. APIs +> finalized, will be added to the AWS CDK with a specific suffix: `BetaX`. APIs > with the preview suffix will never be removed, instead they will be deprecated > and replaced by either the stable version (without the suffix), or by a newer > preview version. For example, assume we add the method -> `grantAwesomePowerPre1`: +> `grantAwesomePowerBeta1`: > > ```ts > /** > * This methods grants awesome powers > */ -> grantAwesomePowerPre1(); +> grantAwesomePowerBeta1(); > ``` > > Times goes by, we get feedback that this method will actually be much better > if it accept a `Principal`. Since adding a required property is a breaking -> change, we will add `grantAwesomePowerPre2()` and deprecate -> `grantAwesomePowerPre1`: +> change, we will add `grantAwesomePowerBeta2()` and deprecate +> `grantAwesomePowerBeta1`: > > ```ts > /** @@ -144,1054 +70,296 @@ AWS CDK v2 release notes: > * > * @param grantee The principal to grant powers to > */ -> grantAwesomePowerPre2(grantee: iam.IGrantable) +> grantAwesomePowerBeta2(grantee: iam.IGrantable) > > /** > * This methods grants awesome powers -> * @deprecated use grantAwesomePowerPre2 +> * @deprecated use grantAwesomePowerBeta2 > */ -> grantAwesomePowerPre1() +> grantAwesomePowerBeta1() > ``` > > When we decide its time to graduate the API, the latest preview version will > be deprecated and the final version - `grantAwesomePower` will be added. > -> #### Alpha modules +> **Alpha modules** > > Writing the perfect API is hard, some APIs will require many iterations of > breaking changes before they can be finalized, others may need a long bake -> time, and some both. This is especially true when writing a new L2. To make -> sure we don't make it too hard to contribute new L2s (keep em coming!), we are -> announcing the `AWS CDK Alpha modules` repo. This repo will be the home of AWS -> CDK modules which are in a very early stage of development. The alpha repo -> will contain multiple modules, each with its own prerelease version. When an +> time, and some both. This is especially true when writing new constructs. +> To that end, new services and L2s will be initially released +> as independent modules, each with their own prerelease versions. When an > alpha module is ready for prime time it will be added to `aws-cdk-lib`. Check -> out our contribution guide for more details. If you are writing a new L2, and -> you are not sure where it should be added, open a GitHub issue in the AWS CDK -> main repo. - -### Modules lifecycle - -This section discuss adding a new modules to `aws-cdk-lib`. Alpha and beta are -optional, modules can be added directly to `aws-cdk-lib` as stable. - -#### Alpha (optional) - -This stage is intended for modules that are in early development stage, they -change rapidly and may incurs many breaking changes. These modules will be -released separately from `aws-cdk-lib` and will have their own version, which -allows for breaking changes in accordance to semver, e.g `0.x` or prerelease -qualifiers. - -In order to release a module as alpha, the following conditions must be met: - -1. Module from `aws-cdk-lib` **can not** depend on it. -2. Can not depend on any other alpha module **\*\***. - -The purpose of the first condition is to prevent cyclic dependencies. To -illustrate the purpose of the second condition lets look at an example. Assume -we have an alpha module A, which depends on an alpha module B, graduating module -A means it will be added to `aws-cdk-lib`, since module A depends on module B, -it means that adding module A to `aws-cdk-lib` breaks condition 1 and creates a -cyclic dependency. To prevent the cycle we will have to graduate B as well, -which is not necessarily what we want. Additionally, if module B is an alpha -module, module A will have to declare a fixed dependency on it, locking its -consumers to the same version of module B - defeating the purpose of -peerDependencies. - -Condition 2 is marked with **\*\*** to note that there might be cases in which -it will make sense for unstable modules to depend on other unstable modules. -This is also evident from he fact that are several such dependencies in v1, see -[appendixes](#unstable-modules-depending-on-other-unstable-modules) for a full -list. - -Such cases will be considered on a per case basis. - -#### Beta (optional) - -In this stage modules are added to `aws-cdk-lib` with a preview suffix in the -name of the module, e.g `aws-batch-beta-x`. Modules in this stage are not -allowed to introduce any breaking changes to their API. Any non backward -compatible change will be introduced via deprecation\*\*. - -> \*\* _The deprecation process will be discussed in the API previews -> specification_ - -#### Stable - -In this stage modules are added to `aws-cdk-lib` under their final name, e.g -`aws-batch`. Addition of new APIs should follow the API previews specification. - -#### Multiple alpha modules VS. single module - -The question of whether we should release experimental modules under one package -or as separate modules has been discussed in this RFC. See -[Option 3](#option-3-separate-multiple-unstable-packages), and -[option 2](#option-2-separate-single-unstable-package). - -I suggest we release alpha modules as separate modules -([Option 3](#option-3-separate-multiple-unstable-packages)). The disadvantages -of this approach, listed in this RFC are: - -> 1. It's not possible for stable modules to depend on unstable ones (see -> Appendix B for data on how necessary that is for the CDK currently), with -> the same implications as above. -> 2. It's not possible for unstable modules to depend on other unstable modules -> (see Appendix B for data on how necessary that is for the CDK currently), -> as doing that brings us back to the dependency hell that mono-CDK was -> designed to solve. -> 3. Graduating a module to stable will be a breaking change for customers. We -> can mitigate this downside by keeping the old unstable package around, but -> that leads to duplicated classes. - -1 and 3 applies to both option 3 and option 2. - -Releasing each alpha module separately has the advantage of allowing consumers -to choose when they want to upgrade a specific module. If we release all modules -under a single package, in which every release may include breaking changes to -any numbers of its modules, users are forced to accept the breaking changes in -all modules, even if they only want to upgrade a single module. For example, -assume the uber package includes `aws-appmesh` and `aws-synthetics`, and that -release `0.x` of the uber package includes breaking changes to both modules, a -customer who only wants the new feature added to `aws-synthetics` now must -accept the breaking changes to `aws-appmesh`, which might include changes to -both the code and their deployed infrastructure they are not ready to make, e.g -resource replacement. Having separate module means reduces the blast radius of -every update. - -As for 2, if an unstable module need to depend on another unstable module, it -will do so using -[peerDependencies](https://docs.npmjs.com/cli/v6/configuring-npm/package-json#peerdependencies). -The "dependency hell" of v1 was a result of using dependencies where we should -have been using peerDependencies. `dependencies` will lead to multiple copies of -a library, `peerDependencies` will not. This subject has been discussed in -length during the RFC review, you can read more in this -[comment](https://github.com/aws/aws-cdk-rfcs/pull/279#discussion_r553581183). - -#### Migrating experimental modules to V2 - -Before v2 release we will need to decide the lifecycle stage of every v1 -experimental module. We will review all modules and devise a migration strategy -in a separate doc - -### Discussion - -Advantages: - -- Preview APIs can be used without declaring a fixed version on `aws-cdk-lib`. -- Since old versions of an API will only be deprecated and not removed, if - needed, we will be able to push critical updates to these versions as well. - For example, if we discover that `grantWrite` grants overly permissive - permissions, and the same occur in all of its experimental (deprecated) - predecessors, we will be able to push the fix to them as well. This will allow - us to get the fix to more customers in case of a critical fix. -- Libraries will be able to use preview APIs without locking their consumers to - a specific version of `aws-cdk-lib` (or any other module we vend). -- Same solution across all languages. - -Disadvantages: - -- Graduating a module will require a lot of code changes from our users to - remove the prefix/suffix. -- User does not have a clear motivation to upgrade to a newer version. -- Low adoption. Users might be hesitant to use an API with a `PreX` in its name, - assuming it means "will break". -- Cluttering the code base. Although most IDEs will mark deprecated properties - with a ~~strikethrough~~, they will still be listed by autocomplete. -- Will force some pretty long and ugly names on our APIs. Many previews APIs - will result in a less aesthetic user code. -- A lot of deprecated code in aws-cdk-lib, possibly blowing up `aws-cdk-lib`. - This might not be a real concern as we can reuse a lot the code between - different version of an API. - -**How can we encourage users to upgrade to a newer version?** - -When a new minor version introduces a breaking change to an API, users have a -clear motivation to upgrade to the new version of the API - they must do so in -order to upgrade to a newer version of the AWS CDK. If the API is only -deprecated, users have no motivation to upgrade to a newer version of the API, -even worse, they might not be aware that a newer version exists. One way to -encourage users to upgrade to a newer version of an API, is to supply tools that -will inform users. For example, we can add a capability to the `cdk doctor` -command that will notify users that their CDK application is using deprecated -experimental APIs. - -Executing `cdk doctor` will print: - -```bash -neta@dev/my-cdk-application$ cdk doctor - -Newer versions of APIs previews your application is using are available. You should consider upgrading. -To see which previews APIs can be upgraded, execute `cdk --show-deprecated-api-usage` -``` - -**If there are no breaking changes, are these APIs really experimental?** - -Given that preview APIs are safe to use, and no breaking changes will be -introduced to them, we might consider not referring to them in any special way, -and if needed, simply add a version to the API name. The first version of -`grantWrite` will be named `grantWrite`, the second version will be named -`grantWriteV1` and so on. While a preview API will not break, we should still -use a naming scheme that convey its non-final nature for the following reasons: - -1. **Real-estate**: `grantWrite` is a much better name than `grantWritePre3` for - the final API. -2. **Encourage feedback**: Declaring an API as "in preview" encourage the - community to supply feedback. -3. **Setting the right expectations**: When an API in preview users are aware - that this is not the final version of the API, and its deprecation is - expected. - -## 3. Extra unstable precautions - -This chapter discusses additional precautions we can choose to implement to -re-inforce goal #1 above. These are orthogonal to the decision on how to divide -the stable and unstable modules (meaning, we could implement any of these with -each of the options above). - -These could be added to either `@experimental` APIs in stable modules, to all -APIs in unstable modules, or both. - -### Require a feature flag for unstable code - -In this variant, we would add a runtime check into all unstable APIs that -immediately fails with an exception if the following context is missing: - -```json -{ - "context": { - "@aws-cdk:allowExperimentalFeatures": true - } -} -``` - -Note that `cdk init` will create a project with this context value set to -`false`. - -To avoid the manual and error-prone process of adding this check to every single -unstable API, we will need to modify JSII so that it recognizes the -`@experimental` decorator, and adds this check during compilation. - -Advantages: - -- Changing the context flag will be an explicit opt in from the customer to - agree to use unstable APIs. - -Disadvantages: - -- This will force setting the flag also for transitive experimental code (for - example, when an unstable API is used as an implementation detail of a - construct, but not in its public interface), which might be confusing. -- Since there is a single flag for all unstable code, setting it once might hide - other instances of using unstable code, working against stated goal #1. -- Requires changes in JSII. - -### Force a naming convention for unstable code - -We can modify `awslint` to force a certain naming convention for unstable code, -for example to add a specific prefix or suffix to all unstable APIs. - -Advantages: - -- Should fulfill goal #1 - it will be impossible to use an unstable API by - accident. -- Does not require changes in JSII, only in `awslint`. - -Disadvantages: - -- Will force some pretty long and ugly names on our APIs. -- Graduating a module will require a lot of code changes from our customers to - remove the prefix/suffix. - -# Appendix A - why can't we break backwards compatibility in the code of mono-CDK main modules? - -This section explains why it will not be possible to break backwards -compatibility of any API inside the stable modules of mono-CDK. - -Imagine we could break backwards compatibility in the code of the `aws-cdk-lib` -main modules. The following scenario would then be possible: - -Let's say we have a third-party library, `my-library`, that vends `MyConstruct`. -It's considered stable by its author. However, inside the implementation of -`MyConstruct`, it uses an experimental construct, `SomeExperiment`, from -mono-CDK's S3 module. It's just an implementation detail, though; it's not -reflected in the API of `MyConstruct`. - -`my-library` is released in version `2.0.0`, and it has a peer dependency on -`aws-cdk-lib` version `2.10.0` (with a caret, so `"aws-cdk-lib": "^2.10.0"`). - -Some time passes, enough that `aws-cdk-lib` is now in version `2.20.0`. A CDK -customer wants to use `my-library` together with the newest and shiniest -`aws-cdk-lib`,`2.20.0`, as they need some recently released features. However, -incidentally, in version `2.15.0` of `aws-cdk-lib`, `SomeExperiment` was broken --- which is fine, it's an experimental API. Suddenly, the combination of -`my-library` `2.0.0` and `aws-cdk-lib` `2.20.0` will fail for the customer at -runtime, and there's basically no way for them to unblock themselves other than -pinning to version `2.14.0` of `aws-cdk-lib`, which was exactly the problem -mono-CDK was designed to prevent in the first place. - -# Appendix B - modules depending on unstable modules +> out our contribution guide for more details. -This section contains the snapshot of the interesting dependencies between -Construct Library modules as of writing this document. +### Developer Guide -## Stable modules depending on unstable modules +The following is a snippet from the Developer Guide targeted at how to install and use the new alpha modules: -``` -⚠️ Stable module '@aws-cdk/aws-applicationautoscaling' depends on unstable module '@aws-cdk/aws-autoscaling-common' -⚠️ Stable module '@aws-cdk/aws-autoscaling' depends on unstable module '@aws-cdk/aws-autoscaling-common' -⚠️ Stable module '@aws-cdk/aws-events-targets' depends on unstable module '@aws-cdk/aws-batch' -⚠️ Stable module '@aws-cdk/aws-lambda' depends on unstable module '@aws-cdk/aws-efs' -⚠️ Stable module '@aws-cdk/aws-stepfunctions-tasks' depends on unstable module '@aws-cdk/aws-batch' -⚠️ Stable module '@aws-cdk/aws-stepfunctions-tasks' depends on unstable module '@aws-cdk/aws-glue' -``` - -## Unstable modules depending on other unstable modules - -``` -ℹ️️ Unstable package '@aws-cdk/aws-apigatewayv2-integrations' depends on unstable package '@aws-cdk/aws-apigatewayv2' -ℹ️️ Unstable package '@aws-cdk/aws-appmesh' depends on unstable package '@aws-cdk/aws-acmpca' -ℹ️️ Unstable package '@aws-cdk/aws-backup' depends on unstable package '@aws-cdk/aws-efs' -ℹ️️ Unstable package '@aws-cdk/aws-docdb' depends on unstable package '@aws-cdk/aws-efs' -ℹ️️ Unstable package '@aws-cdk/aws-ses-actions' depends on unstable package '@aws-cdk/aws-ses' -``` - -# Appendix C - discarded solutions to problem #2 - -These potential solutions to problem #2 were discarded by the team as not -viable. - -## Option 1: separate submodules of `aws-cdk-lib` - -In this option, we would use the namespacing features of each language to vend a -separate namespace for the experimental APIs. The customer would have to -explicitly opt-in by using a language-level import of a namespace with -"experimental" in the name. - -Example using stable and unstable Cognito APIs: - -```ts -import * as cognito from 'aws-cdk-lib/aws-cognito'; -import * as cognito_preview from 'aws-cdk-lib/experimental/aws-cognito'; - -const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); -const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; -const userPoolClient = new cognito.UserPoolClient(...); -``` - -Advantages: - -1. It's possible for stable module to depend on unstable ones (see Appendix B - for data on how necessary that is for the CDK currently) -2. It's possible for unstable modules to depend on other unstable modules (see - Appendix B for data on how necessary that is for the CDK currently). - -Disadvantages: +> **Installing the Alpha CDK Modules** +> +> Alpha CDK modules are denoted with an alpha identifier, to clearly identify them as pre-production. The following examples will walk +> through installing the module for a hypothetical AWS service called FooBar. +> +> *Typescript/Javascript* +> +> ```sh +> npm install @aws-cdk/aws-foobar-alpha +> ``` +> +> *Python* +> +> ```sh +> pip install aws-cdk.aws-foobar-alpha +> ``` +> +> *Java* +> +> Add the following to the `` container of pom.xml. +> +> ```xml +> +> software.amazon.awscdk +> foobar-alpha +> ${version} +> +> ``` +> +> *Dotnet* +> +> ```sh +> dotnet add package Amazon.CDK.AWS.FooBar.Alpha +> ``` +> +> *Go* +> +> ```sh +> go get github.com/aws/aws-cdk-go/awscdk/v2/awsfoobaralpha +> ``` +> +> **Using the Alpha CDK Modules** +> +> The following examples show how to import the FooBar service into your code. Imports for the core library and S3 are shown for comparison. +> +> *Typescript/Javascript* +> +> ```ts +> import { App, Stack } from 'aws-cdk-lib'; +> import { aws_s3 as s3 } from 'aws-cdk-lib'; +> import * as foobar from '@aws-cdk/aws-foobar-alpha'; +> ``` +> +> *Python* +> +> ```python +> from aws_cdk import App, Stack +> from aws_cdk import aws_s3 as s3 +> from aws_cdk import aws_foobar_alpha as foobar +> ``` +> +> *Java* +> +> ```java +> import software.amazon.awscdk.App; +> import software.amazon.awscdk.Stack; +> import software.amazon.awscdk.services.s3.Bucket; +> import software.amazon.awscdk.services.foobar.alpha.FooBarConstruct; +> ``` +> +> *Dotnet* +> +> ```csharp +> using Amazon.CDK; +> using Amazon.CDK.AWS.S3; +> using Amazon.CDK.AWS.FooBar.Alpha; +> ``` +> +> *Go* +> +> ```go +> import ( +> "github.com/aws/aws-cdk-go/awscdk" +> "github.com/aws/aws-cdk-go/awscdk/v2/awss3" +> "github.com/aws/aws-cdk-go/awscdk/v2/awsfoobaralpha" +> ) +> ``` +> +> **Versioning** +> +> Alpha modules are released separately from `aws-cdk-lib`, but their versioning mirrors that of `aws-cdk-lib`. +> For each release of `aws-cdk-lib` (e.g., `2.x.y`), the latest version of all of the alpha modules will also be +> released, with a corresponding `alpha` pre-release version (e.g., `2.x.y-alpha.0`). Generally, using versions +> of alpha modules that match the `aws-cdk-lib` version ensures compatibility; however, you can also use a newer +> version of `aws-cdk-lib` than the version of the alpha modules, allowing you to get new features from `aws-cdk-lib` +> without needing to also take on new (potentially breaking) changes from the alpha modules. -1. Might be considered less explicit, as a customer never says they want to - depend on a package containing unstable APIs, or with `0.x` for the version. -2. If a third-party package depends on an unstable API in a non-obvious way (for - example, only in the implementation of a construct, not in its public API), - that might break for customers when upgrading to a version of `aws-cdk-lib` - that has broken that functionality compared to the `aws-cdk-lib` version the - third-party construct is built against (basically, the same scenario from - above that explains why we can no longer have unstable code in stable - mono-CDK modules). None of the options solve the problem of allowing - third-party libraries to safely depend on unstable Construct Library code; - however, the fact that all unstable code in this variant is shipped in - `aws-cdk-lib` makes this particular problem more likely to manifest itself. -3. Graduating a module to stable will be a breaking change for customers. We can - mitigate this downside by keeping the old unstable module around, but that - leads to duplicated classes in the same package. +### CHANGELOG & Release Notes -**Verdict**: discarded because disadvantage #2 was considered a show-stopper. +The main V2 Changelog (`CHANGELOG.v2.md`) will include only changes from stable modules, in the standard format. A +separate Changelog (`CHANGELOG.v2.alpha.md`) will be created to track all changes to alpha modules. -## Option 4: separate V3 that's all unstable +**CHANGELOG.v2.md:** -In this option, we will fork the CDK codebase and maintain 2 long-lived -branches: one for version `2.x`, which will be all stable, and one for version -`3.x`, which will be all unstable. +```md +## [2.1.0](https://github.com/aws/aws-cdk/compare/v2.0.0...v2.1.0) (2022-01-01) -Example using stable and unstable Cognito APIs: (assuming the dependency on -`"aws-cdk-lib"` is in version `"3.x.y"`): +### Features -```ts -import * as cognito from 'aws-cdk-lib/aws-cognito'; +* **bar:** new fizzbuzz support ([#999999](https://github.com/aws/aws-cdk/issues/999999)) (b01dface), closes [#999998](https://github.com/aws/aws-cdk/issues/999998) +* **foo:** more buzzing on the fizzes ([#999994](https://github.com/aws/aws-cdk/issues/999999)) (deadb33f), closes [#999993](https://github.com/aws/aws-cdk/issues/999993) -const idp = new cognito.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); -const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; -const userPoolClient = new cognito.UserPoolClient(...); -``` +### Bug Fixes -Advantages: - -1. It's possible for unstable modules to depend on other unstable modules (see - Appendix B for data on how necessary that is for the CDK currently). - -Disadvantages: - -1. It's not possible for stable modules to depend on unstable ones (with the - same implications as above). -2. Does not make it obvious to customers that this is unstable (`3.x` is - considered stable in semantic versioning). -3. We are going from "some code is stable, some is unstable" to "all of this is - unstable", which seems to be against the customer feedback we're hearing - that's the motivation for this RFC. -4. Two long-lived Git branches will mean constant merge-hell between the two, - and since `3.x` has free rein to change anything, there will be a guarantee - of constant conflicts between the two. -5. Fragments the mono-CDK third-party library community into two. -6. Very confusing when we want to release the next major version of the CDK (I - guess we go straight to `4.x`...?). -7. The fact that all code in `3.x` is unstable means peer dependencies don't - work (see above for why). -8. Graduating a module to stable will be a breaking change for customers. We can - mitigate this downside by keeping the old unstable package around, but that - leads to duplicated classes between the 2 versions. - -**Verdict**: discarded as a worse version of option #5. - -### Option 2: separate single unstable package - -Instead of vending the unstable modules together with the stable ones, we can -vend a second mono-CDK, `aws-cdk-lib-experiments` (actual name can be changed -before release of course). A customer will have to explicitly depend on -`aws-cdk-lib-experiments`, which will be released in version `0.x` to make it -even more obvious that this is unstable code. `aws-cdk-lib-experiments` would -have a caret peer dependency on `aws-cdk-lib`. - -Example using stable and unstable Cognito APIs: - -```ts -import * as cognito from 'aws-cdk-lib/aws-cognito'; -import * as cognito_preview from 'aws-cdk-lib-experiments/aws-cognito'; - -const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); -const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; -const userPoolClient = new cognito.UserPoolClient(...); +* **core:** transmorgrifier sometimes clones subject ([#999991](https://github.com/aws/aws-cdk/issues/15313)) (0ddba11), closes [#999990](https://github.com/aws/aws-cdk/issues/999990) ``` -Advantages: - -1. Very explicit (customer has to add a dependency on a package with - "experiments" in the name and version `0.x`). -2. It's possible for unstable modules to depend on other unstable modules (see - Appendix B for data on how necessary that is for the CDK currently). - -Disadvantages: +**CHANGELOG.v2.alpha.md:** -1. It's not possible for stable modules to depend on unstable ones (see Appendix - B for data on how necessary that is for the CDK currently). This has serious - side implications: +```md +## [2.1.0-alpha.0](https://github.com/aws/aws-cdk/compare/v2.0.0-alpha.0...v2.1.0-alpha.0) (2022-01-01) - - All unstable modules that have stable dependents today will have to be - graduated before `v2.0` is released. - - Before a module is graduated, all of its dependencies need to be graduated. - - It will not be possible to add new dependencies on unstable modules to - stable modules in the future (for example, that's a common need for - StepFunction Tasks). +### BREAKING CHANGES +* **newbar:** default answer to life, universe and everything changed from 41 to 42. -2. Graduating a module to stable will be a breaking change for customers. We can - mitigate this downside by keeping the old unstable module around, but that - leads to duplicated classes. +### Features -### Option 3: separate multiple unstable packages +* **newbar:** add support for adding foos ([#999999](https://github.com/aws/aws-cdk/issues/999999)) (b01dface), closes [#999998](https://github.com/aws/aws-cdk/issues/999998) -In this option, each experimental library will be vended as a separate package. -Each would have the name "experiments" in it (possible naming convention: -`@aws-cdk-lib-experiments/aws-`), and would be released in version -`0.x` to make it absolutely obvious this is unstable code. Each package would -declare a caret peer dependency on `aws-cdk-lib`. +### Bug Fixes -Example using stable and unstable Cognito APIs: - -```ts -import * as cognito from 'aws-cdk-lib/aws-cognito'; -import * as cognito_preview from '@aws-cdk-lib-experiments/aws-cognito'; - -const idp = new cognito_preview.UserPoolIdentityProviderOidc(this, 'OIDC', { ... }); -const supported = [cognito.UserPoolClientIdentityProvider.custom("MyProviderName")]; -const userPoolClient = new cognito.UserPoolClient(...); +* **newbar:** answer to life has off-by-one error ([#999991](https://github.com/aws/aws-cdk/issues/15313)) (0ddba11), closes [#999990](https://github.com/aws/aws-cdk/issues/999990) ``` -Advantages: - -1. Very explicit (customer has to add a dependency on a package with - "experiments" in the name and version `0.x`). -2. This is closest to the third-party CDK package experience our customers will - have. - -Disadvantages: - -1. It's not possible for stable modules to depend on unstable ones (see Appendix - B for data on how necessary that is for the CDK currently), with the same - implications as above. -2. It's not possible for unstable modules to depend on other unstable modules - (see Appendix B for data on how necessary that is for the CDK currently), as - doing that brings us back to the dependency hell that mono-CDK was designed - to solve. -3. Graduating a module to stable will be a breaking change for customers. We can - mitigate this downside by keeping the old unstable package around, but that - leads to duplicated classes. - -### Option 5: superset/subset releases - -Instead of _splitting_ the stable and experimental APIs into two different -packages, we release two builds of the same source code, where the -_experimental_ build is a superset of the _stable_ build. - -This can be done automatically, by using jsii to strip out experimental APIs -from the **public interface** of the stable release. The implementation (`.js` -files) will be the same in both releases, so stable APIs in `aws-cdk-lib` can -safely use experimental APIs internally (though not in their public API -surfaces). - -#### How does stripping APIs from the public work? - -| TypeScript | JavaScript | Java, Python, C#, Go | -| --------------------- | ---------- | ------------------------------- | -| Stripped from `.d.ts` | N/A | No bindings generated by `jsii` | - -In TypeScript, the compiler will prevent usage of APIs that are not advertised -in the `.d.ts` files. Users will be able to bypass this guarantee by using -`as any` though. - -For JavaScript, an IDE will (hopefully¹) not autocomplete APIs missing from the -`.d.ts` files; otherwise nothing prevents users from calling them. On the other -hand, JavaScript devs are by definition required to read the docs, so they can't -miss the _experimental_ banners displayed there. - -For jsii client languages, the bindings for experimental APIs will simply not -exist so there's no way to work around that (though we'll have to check whether -for example Python makes it possible to pass struct values that aren't in the -declared API surface). - -¹) A _smart enough_ IDE could discover those APIs by parsing the JavaScript -directly, or by doing dynamic execution and runtime inspection of objects. This -was already being done before TypeScript existed, I did not survey the current -landscape of IDEs to figure out which ones use what techniques to provide -autocomplete for users. - -#### What about experimental struct properties - -We also commonly use `@experimental` to indicate experimental struct properties -for `Props` types. - -There will be code that users can write that will pass a props object that -technically contains non-advertised, experimental properties. Without additional -runtime support the object constructor will happily interpret the presence of -those properties and make decisions based off of them. - -An example in TypeScript: - -```ts -// Construct -interface StableConstructProps { - readonly stableProp: string; - - /** @experimental */ - readonly experimentalProp: string; -} - -class StableConstruct { - constructor(scope: Construct, id: string, props: StableConstructProps) { - super(scope, id); - if (props.experimentalProp) { - console.log('Experimental features activated!'); - } - } -} - -//---------------------------------------------------------------------- -// THE FOLLOWING CLIENT CODE IS USING THE STABLE BUILD -// -// As far as this client code is aware, StableConstructProps looks like this: -// -// interface StableConstructProps { -// readonly stableProp: string; -// } - -const props = { - stableProp: 'foo', - experimentalProp: 'bar', -}; - -// Even though we're using "stable mode", the following code still -// prints 'Experimental features activated!'. -// -// The call below is allowed because 'typeof props' is a superset of StableConstructProps. -new StableConstruct(this, 'Id', props); - -// In contrast the call below would NOT be allowed: -new StableConstruct(this, 'Id', { - stableProp: 'foo', - experimentalProp: 'bar', -}); -``` - -We would need runtime support -(`if (props.experimentalProp && IN_EXPERIMENTAL_MODE) { ... }`) to detect and -prevent this, or accept it. - -The example above is for TypeScript; it obviously also holds for JavaScript, and -_might_ hold for jsii languages in certain cases. - -#### Vending - -There are a number of different ways we can choose to distinguish the two -builds. See Appendix D for an evaluation of all the possibilities. - -- Distinguish by package name - - Can't be used in **pip**: there's no way to have an app that uses the - experimental library work with a library that uses the stable library (no - way to prevent the transitive dependency package with a different name from - being installed). -- Distinguish by prerelease tag - - **NuGet**: requires that experimental version semver-sorts _after_ the - matching stable version, otherwise it errors out and fails the restore - operation. - - **npm**: will always complain when trying to satisfy a stable requirement - with an experimental version, regardless of how it sorts. - - **Maven**: does not recognize prerelease tags. This means that _apps_ can't - have an open-ended version range since they might accidentally pick up - experimental versions (a Maven range of `[1.60.0, 2.0.0)` _will_ match - `1.61.0-experimental`). This might not be a problem as this is rarely done - in practice (since Maven doesn't have a lock file, the `pom` itself serves - as the lock file). -- Distinguish by major version - - Very similar to prerelease tags, except we don't depend on Package Manager's - support for prerelease tags; all PMs support major versions. - -Even though there are downsides, the prerelease tag is the least bad of the -alternatives and the one that's (probably) easiest to explain. - -We have to work around the version ordering problem though, by making sure the -experimental version sorts after the stable version. We can either use the minor -or the patch version: - -```shell -# Experimental uses minor bump -1.60.0 < 1.61.0-experimental - -# Experimental uses (fake) patch bump -1.60.0 < 1.60.100-experimental -``` - -#### Summary - -Advantages: - -1. It's possible for stable modules to depend on unstable ones (see Appendix B - for data on how necessary that is for the CDK currently). -2. Stabilizing code will not be a breaking change for customers. -3. Stabilizing code will be a simple operation for CDK developers (remove an - annotation). -4. No additional management overhead of multiple packages. -5. It's possible for unstable modules to depend on other unstable modules. - -Disadvantages: - -1. Stable/experimental versioning scheme is not based on any well-known industry - standard, we're going to have to clearly explain it to people. -2. Requires changes to jsii -3. Requires changes to the docs to make switching between editions possible -4. Protection not offered for JavaScript users, can be bypassed in TypeScript; - although something is to be said for it being the same with `private` fields - today. -5. Does not satisfy - [goal 4 - Using experimental modules should be easy](#4-using-experimental-modules-should-be-easy). - Customers that wants to use a single experimental API must pay the cost of - using a different version of the **entire** `aws-cdk-lib`. This is a non - trivial cost due to the following: - - Migrating to either a new major version, or a prerelease version, is a - process that usually includes accepting lot of breaking changes. While this - may not be the case for `aws-cdk-lib`, users wil be still hesitant as it is - the standard meaning of such migration. - - Users which uses the experimental version might use experimental APIs they - didn't intend to, similar to the experimental experience in v1. - -#### POC results - -> _Option 5 should be move to the appendices section, it is added here for the -> purpose of minimizing the diff in the review process. It will be moved once -> the RFC is finalized._ - -Option 5 was rejected since there is no way to model the relationship between -the "experimental" version and the "stable" version through semantic versioning. - -##### Prerelease qualifiers - -According to the semver [specification](https://semver.org/#spec-item-11), a -precedence between versions is calculated **only** if the **major**, **minor** -and **patch** are equal. This means that there is no way for a constructs -library to declare a range of supported versions which include an experimental -version. Sticking to the above example, a CDK construct library declaring a peer -dependency on version `^1.60.0` of `aws-cdk-lib`, can not be used in a CDK -application that declares a dependency on `aws-cdk-lib` version -`1.61.0-experimental`. This is because the patch part in `1.60.0` and -`1.61.0-experimental` is not equal, which means `1.61.0-experimental` does not -satisfy `^1.60.0`. In npm versions prior to npm-v7, if `peerDependencies` -requirements are not met, executing `npm install` would only issue a warning. In -npm-v7, which automatically tries to install `peerDependencies`, executing -`npm install` will throw an error. - -To illustrate the user experience with npm-v7. The below is the output of -executing `npm install` in a CDK application (`my-cdk-application`), which -declares a dependency on a `aws-cdk-lib` version `1.61.0-experimental`, and on a -CDK library (`my-construct-lib`) which itself declares a peer dependency on -`aws-cdk-lib` version `^1.60.0`: - -``` -npm ERR! code ERESOLVE -npm ERR! ERESOLVE unable to resolve dependency tree -npm ERR! -npm ERR! While resolving: my-cdk-application@1.0.0 -npm ERR! Found: aws-cdk-lib@1.61.0-experimental -npm ERR! node_modules/aws-cdk-lib -npm ERR! aws-cdk-lib"@1.61.0-experimental" from the root project -npm ERR! -npm ERR! Could not resolve dependency: -npm ERR! peer aws-cdk-li@"^1.60.0" from my-construct-lib@1.6.0 -npm ERR! node_modules/my-construct-lib -npm ERR! my-construct-lib@"1.6.0" from the root project -npm ERR! -npm ERR! Fix the upstream dependency conflict, or retry -npm ERR! this command with --force, or --legacy-peer-deps -npm ERR! to accept an incorrect (and potentially broken) dependency resolution. -npm ERR! -``` - -Users can work around this by executing `npm install --force` which means that -npm will not check for version compatibility at all, and therefore not an -acceptable solution. - -##### Separate major versions - -This was rejected due to similar reasons as listed in the disadvantages of -[Option 4: separate V3 that's all unstable](#option-4-separate-v3-thats-all-unstable). - -# Appendix D - Alternatives for identifying stable/experimental builds - -We have the following options to pick from: - -- Different package names (`aws-cdk-lib` vs `aws-cdk-lib-experimental`). -- Pre-release version tag -- Different major versions - -We need to evaluate all of these, for each package manager, on the following -criteria: - -- Can applications depend on an upwards unbounded range of _stable_ builds? - (important) -- Can an application using the _experimental_ build use a library using the - _stable_ build? (important) - - Without the package manager complaining (preferably) -- Can a library declare a tool-checked dependency on a specific _experimental_ - version? (preferably) - -We have ruled out the requirement that an application using _stable_ may -transparently use a library using _experimental_. We will disallow this as it's -not feasible. We're trying to achieve the following compatibility matrix: - -| . | Stable App | Experimental App | -| ---------------- | ----------------- | ------------------------ | -| Stable Lib | floating versions | lib floating, app pinned | -| Experimental Lib | - | pinned versions | - -> NOTE: in the following sections, I'll be using the terms "stable app", -> "experimental app", "stable lib" and "experimental lib" as stand-ins for the -> more accurate but much longer "app that depends on stable CDK", "app that -> depends on experimental CDK", etc. - -## Different package names - -We would vend two different package names, for example: - -- `aws-cdk-lib` -- `aws-cdk-lib-experimental` - -The tl;dr of this section is as follows. Read below for details. - -| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | -| ----- | ----------- | -------------------------------- | --------------------- | --------------------------- | -| NPM | yes | with package aliases | yes | no | -| Maven | yes | with excluded dependencies | yes | yes | -| pip | yes | no | - | no | -| NuGet | yes | no | no | yes | -| Go | ? | ? | | ? | +**Github Release Notes:** -### NPM\* +The release notes (e.g., ) for each release will contain the combined notes for both +stable and alpha modules, clearly delineated. -In NPM-based languages (JavaScript and TypeScript), the package name appears in -source code (in the `require()` statement). If we changed the library name, we -wouldn't be able to use a library that uses stable code (it would contain -`require('aws-cdk-lib')`) in an application that uses experimental code: - -```js -//------------- LIBRARY THAT USES STABLE --------------- -// package.json -{ - "peerDependencies": { - "aws-cdk-lib": "^1.60.0" - } -} - -// index.ts -import * as cdk from 'aws-cdk-lib'; // <- locally perfectly sane - -//------------- APPLICATION THAT USES EXPERIMENTAL AND THIS LIB --------------- -// package.json -{ - "devDependencies": { - "lib-that-uses-stable": "^1.2.3", - "aws-cdk-lib-experimental": "1.60.0" - } -} - -// app.ts -import * as cdk from 'aws-cdk-lib-experimental'; // <- locally perfectly sane -``` - -Well oops. Turns out we can't have a library that depends on `aws-cdk-lib` in an -application that uses `aws-cdk-lib-experimental`; NPM will complain about the -missing `peerDependency`, and the `require('aws-cdk-lib')` call will just fail -at runtime. - -NPM 6.9.0 (released May 2019) introduces a feature that helps with this: -[package aliases](https://github.com/npm/rfcs/blob/latest/implemented/0001-package-aliases.md) -which allows installing a package under a different name, so we could install -`aws-cdk-lib-experimental` under the name `aws-cdk-lib`. The application -`package.json` will now look like this: - -```js -//------------- APPLICATION THAT USES EXPERIMENTAL AND STABLE LIB --------------- -// package.json -{ - "devDependencies": { - "lib-that-uses-stable": "^1.2.3", - "aws-cdk-lib": "npm:aws-cdk-lib-experiments@1.60.0", - } -} - -// app.ts -import * as cdk from 'aws-cdk-lib'; -``` +```md +# [v2.1.0] +--- +## aws-cdk-lib +### Features +* **bar:** new fizzbuzz support ([#999999](https://github.com/aws/aws-cdk/issues/999999)) (b01dface), closes [#999998](https://github.com/aws/aws-cdk/issues/999998) +* **foo:** more buzzing on the fizzes ([#999994](https://github.com/aws/aws-cdk/issues/999999)) (deadb33f), closes [#999993](https://github.com/aws/aws-cdk/issues/999993) -So that's cool. +### Bug Fixes +* **core:** transmorgrifier sometimes clones subject ([#999991](https://github.com/aws/aws-cdk/issues/15313)) (0ddba11), closes [#999990](https://github.com/aws/aws-cdk/issues/999990) -What would a construct library that depends on an experimental build look like, -and could we use that in the same application? +-- +## Alpha Modules (v2.1.0-alpha.0) +### BREAKING CHANGES +* **newbar:** default answer to life, universe and everything changed from 41 to 42. -An experimental library would declare this: +### Features +* **newbar:** add support for adding foos ([#999999](https://github.com/aws/aws-cdk/issues/999999)) (b01dface), closes [#999998](https://github.com/aws/aws-cdk/issues/999998) -```json -{ - "peerDependencies": { - "aws-cdk-lib": "1.60.0" - // Note: ideally you'd like to express "aws-cdk-lib-experiments": "1.60.0" here, but - // if you do that "npm ls" will complain about an unmet peer dependency - }, - "devDependencies": { - "aws-cdk-lib": "npm:aws-cdk-lib-experiments@1.60.0" - } -} +### Bug Fixes +* **newbar:** answer to life has off-by-one error ([#999991](https://github.com/aws/aws-cdk/issues/15313)) (0ddba11), closes [#999990](https://github.com/aws/aws-cdk/issues/999990) ``` -This will all work as intended and put code in the right places. - -However, we lose the tool-checked dependency on experimental: from NPM's point -of view the library depends on _stable_ CDK (whereas we want it to declare that -it depends on _experimental_ CDK). It's only notes in the README that can inform -the consumer otherwise. - -### Maven - -No need to change the sources when using a stable library with an experimental -app, as the namespaces and class names will be the same. - -Using a stable library in an experimental app can be done in two ways: - -- The library uses the `provided` dependency flavor (similar to - `peerDependencies` in NPM). Requires the app to bring the right version of - CDK. -- The app that wants to use the _experimental_ build uses dependency exclusions - to remove the dependency on the _stable_ build (looks like this is necessary - for every stable library individually) - -[Reference](https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html#) - -### pip - -No need to change the sources when using a stable library with an experimental -app, as the namespaces and class names will be the same. - -`pip` transitively fetches from the library's `setup.py`, and there's no way to -override that. - -Our only option around that is to have stable libraries not declare any CDK -dependency in `setup.py`, but then we also lose the ability for libraries to -specify their dependency version, which is not really acceptable. - -Effectively, this seems to be a no-go. - -### .NET Core/NuGet +## FAQ -No need to change the sources when using a stable library with an experimental -app, as the namespaces and class names will be the same. +### *Can I take a dependency on just a single alpha module?* -...pending responses from .NET SDK team... +Yes! Each of the alpha modules are independently published and versioned, so you can install and use only the +alpha module(s) you need for your infrastructure. All other modules (via `aws-cdk-lib`) will offer stable APIs that +will not change in backwards-incompatible ways. -### Go +### *Can I mix module versions (i.e., upgrade only one)?* -??? +Yes, since each alpha module is its own published package, each can be separately upgraded. You can upgrade to +receive new features in one module without needing to upgrade other unrelated alpha modules. -## Prerelease tags +Note that in some cases, alpha modules have dependencies on each other (e.g., `aws-apigatewayv2` and +`aws-apigatewayv2-integrations`); in these cases both modules may need to be upgraded simultaneously. -We would vend the experimental build under a prerelease tag, for example: +### *Do the alpha modules need to be at the same version as aws-cdk-lib?* -- `1.60.0` -- `1.60.0-experimental` +No, unlike CDK v1 independent modules, the v2 alpha modules do not need to exactly match each other or the main +aws-cdk-lib. Each alpha module declares a dependency on a minimum version of aws-cdk-lib required for it to +function; as long as that requirement is satisfied, different versions can be installed. For example, you may have +`aws-cdk-lib@2.4.0` installed with `@aws-cdk/foobar-alpha@2.2.0-alpha.0` and `@aws-cdk/fizzbuzz-alpha@2.3.1-alpha.0`. -Because of semver ordering, if we use the numbering above then the -`experimental` version will sort _before_ the stable version, which may lead to -problems because a library's dependency requirement of `^1.60.0` would _not_ be -satisfied by a version numbered `1.60.0-experimental`. +### *Can I safely update an alpha module without receiving breaking changes?* -We could get rid of some of the ordering and warning problems by making sure the -experimental version semver-orders _after_ the stable version by making sure we -bump some number: +Not automatically. Check the CHANGELOG for notices of breaking changes to the alpha modules. -- `aws-cdk-lib@1.60.100-experimental` -- `aws-cdk-lib@1.61.0-experimental` +### *Where can I view the docs for the alpha modules?* -| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | -| ----- | ------------------ | -------------------------------- | --------------------------- | --------------------------- | -| NPM | yes | yes | no | yes | -| Maven | no but that's okay | yes | yes | yes | -| pip | yes | yes | if experimental sorts later | yes | -| NuGet | yes | if experimental sorts later | yes | yes | -| Go | ? | ? | ? | ? | +The docs for the alpha modules are available on the CDK API documentation website, +. -### NPM +### *Where can I see what’s changed for each of the modules?* -Can apps use stable ranges: yes, `^1.60.0` will not auto-pick -`1.61.1-experimental` for downloading. +An aggregated Changelog is published for all of the alpha modules, located alongside the primary Changelog, and +linked to from our GitHub artifacts (see ). -Experimental app uses stable lib: can be done, the lib only uses a -`peerDependency`. +## Internal FAQ -However, if the `peerDependency` of a library is stable (`^1.60.0`) and we're -trying to satisfy it with an experimental version, `npm ls` will complain -_regardless_ of where the experimental version sorts with respect to the stable -version. +### *How do we tag releases in the repository?* -It will _work_, but npm will _complain_. +We rely on the V2 tags to determine the differences that are part of this release. We do not add tags specifically for +the alpha modules, as the versioning for alpha modules is always incremented at the same time as the V2 versions. -### Maven +### *Where will the V2 Changelogs be stored?* -Stable ranges: no, `[1.60.0,)` _will_ include `1.61.1-experimental`. However, -this might not be an issue as `pom.xml` typically serves as the lock file and -people will not build applications with open-ended ranges in the pom file. +The existing `CHANGELOG.v2.md` Changelog, at the root of the repo, will remain unchanged. The new `CHANGELOG.v2.alpha.md` +file will be stored alongside it, also at the root of the repo. Both Changelogs will only be present on the `v2-main` +branch of the repo. -If we _wanted_ to, we could vend as `1.60.0-experimental-SNAPSHOT`; snapshot -versions will not match a range unless specifically requested. However, -`SNAPSHOT` versions are not intended for this, they are intended to indicate -mutability and bypass caches: every configurable time period (by default, every -day but can be every build, and snapshots can also be disabled altogether) the -`SNAPSHOT` version will be fetchd again, as it is assumed to be mutable. +### *What is the relationship between the aws-cdk-lib version and alpha module versions?* -Maven uses a "closest to root wins" dependency model, so the application can -substitute an experimental version in place of the declared stable -compatibility. +At publishing time, the alpha modules will be given a dependency on the current (corresponding) `aws-cdk-lib` version. +For example, `@aws-cdk/aws-foobar-alpha@2.2.0-alpha.0` will have a dependency on `aws-cdk-lib@2.2.0`. This automatic +version bump is the simplest to implement and requires the lowest amount of extra testing infrastructure to verify +correctness. The alternative -- manually bumping the dependent `aws-cdk-lib` version based on a set of criteria -- +was deemed too complex to be feasible. -It will not complain about incompatible versions unless you really _really_ ask -for diagnostics (and even then it's hard to get it to show an error). +## Appendix -### pip +### Appendix A: Goals -Stable ranges: yes, `>=1.60.0` will not match version `1.60.1-experimental` -[ref](https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions) - -Experimental app uses stable lib: `pip` allows overriding transitive -dependencies using `requirements.txt`. - -It will complain if the experimental version sorts before the stable version, -but will NOT complain if the experimental version sorts after the stable -version. Everything still gets installed, even if it complains. - -Example of a complaint (using a different package): - -```text -awscli 1.18.158 has requirement s3transfer<0.4.0,>=0.3.0, but you'll have s3transfer 0.2.0 which is incompatible. -``` - -(Note: when trying the PyPI testing server, the version number schema I had in -mind: `X.Y.Z-experimental` is not accepted, nor is `X.Y-experimental` or -`X.Yexp`. Ultimately I had to go for `1.60rc1`.) - -### NuGet - -PJ Pittle from the .NET team has confirmed for us that NuGet will complain and -**error out** if we use: - -- Library depends on CDK `>= 1.60.0` -- App uses CDK `1.60.0-experimental` - -(Because of semver ordering.) - -The only solution seems to be to make sure that the experimental version sorts -after the stable version. - -## Different major versions - -We would vend the experimental build under a prerelease tag, for example: - -- `aws-cdk-lib@2.60.0` -- `aws-cdk-lib@3.60.0` (odd ranges are experimental) - -Or: - -- `aws-cdk-lib@2.60.0` -- `aws-cdk-lib@102.60.0` (100+ versions are experimental) - -| . | Stable apps | Experimental app uses Stable lib | ...without complaints | Lib advertises Experimental | -| ----- | ----------- | -------------------------------- | --------------------- | --------------------------- | -| NPM | yes | yes | yes | yes | -| Maven | yes | yes | yes | yes | -| pip | yes | no | - | yes | -| NuGet | yes | yes | ? | yes | -| Go | ? | ? | ? | ? | - -### NPM +These are the goals of this RFC, in order from most to least important: -Mostly like pre-release tags, except libraries using stable can explicitly -declare they're usable with both stable and experimental ranges by using -multiple `peerDependency` ranges: +1. Using CDK APIs that don't guarantee backwards-compatibility should require clear, explicit opt-in -```json -{ - "peerDependencies": { - "aws-cdk-lib": "^2.60.0 ^102.60.0" - } -} -``` + It should be absolutely obvious to a CDK customer when they are opting in to + using an API that might have backwards-incompatible changes in the future. From + experience, we have determined that including that information in the `ReadMe` + file of a module, or in the inline code documentation available in an + editor/IDE, does not meet the criteria of "absolutely obvious". -This would suppress the warning you would otherwise get for a non-compatible -version. +1. We want to foster a vibrant ecosystem of third-party CDK packages -### Maven + In our estimation, the CDK cannot be successful without growing an expansive + collection of third-party packages that provide reusable Constructs on various + levels of abstraction. Changing to vending the Construct Library as a monolithic + package is one part of making that possible; we should make sure our approach to + alpha modules and new experimental code also takes this into account. -Like pre-release tags, except we don't get the mixing in of pre-release versions -with regular versions. +1. The CDK team can still perform API experiments -### pip + We believe that one of the reasons for CDK's success is the ability to release + functionality quickly into the hands of customers, to get their feedback. The + ability to release experiments is crucial for that speed; if every single + released API decision carried with it the absolute burden of being 100% + backwards compatible, that would slow the pace of CDK innovation considerably + (especially third-party contributions), and would lengthen the feedback loop + from our customers on the quality of the proposed APIs. -Like pre-release tags. + For those reasons, we consider it essential for the CDK team to retain the + capability to perform experiments with our APIs (of course, only those that are + clearly marked as such). -### NuGet +1. Using alpha modules should be easy -Like pre-release tags. + Our development methodology is highly dependent on feedback from the community + before finalizing APIs. To encourage users to use and provide feedback on + experimental APIs, we should make them easy to use. -### Go +### Appendix B: Previous RFC -? +This is the second version of this RFC. A previous version was reviewed and approved, and can be found here: +. +The original version is more implementation- and trade-off focused, whereas this RFC focused on the working-backwards artifacts.