Skip to content

Commit

Permalink
chore: upgrade lerna and use nx (#24896)
Browse files Browse the repository at this point in the history
Updating lerna to the latest version which now uses `nx` under the hood. Apart from just updating the version, this also allows us to take advantage of nx caching. For now this has only been enabled locally.

I'm not 100% sure about the `nx` configuration since our repo is a little weird, but I've been running locally with this configuration for probably 6 months and it has worked fine for me.

Side note - the repo restructure has caused some weirdness with our tests. I had to make some updates to existing tests, and I do not know why.

Closes #<issue number here>.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
corymhall authored Apr 13, 2023
1 parent a344507 commit 046d0fc
Show file tree
Hide file tree
Showing 20 changed files with 2,084 additions and 1,772 deletions.
18 changes: 12 additions & 6 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,12 @@ Most contributions only require working on a single package, usually `aws-cdk-li
time, you can execute the following to build it and it's dependencies.

```console
$ cd packages/aws-cdk-lib
$ ../../scripts/buildup
$ npx lerna run build --scope=aws-cdk-lib
```

Note: The `buildup` command is resumable. If your build fails, you can fix the issue and run `buildup --resume` to
resume.
Note: `lerna` uses a local cache by default. If your build fails, you can fix
the issue and run the command again and it will not rerun any previously
successful steps.

At this point, you can run build and test the `aws-cdk-lib` module by running

Expand All @@ -121,13 +121,19 @@ However, if you wish to build the entire repository, the following command will

```console
cd <root of the CDK repo>
scripts/foreach.sh yarn build
npx lerna run build
```
Note: The `foreach` command is resumable by default; you must supply `-r` or `--reset` to start a new session.

You are now ready to start contributing to the CDK. See the [Pull Requests](#pull-requests) section on how to make your
changes and submit it as a pull request.

If you want to run a build without using the local cache, provide the
`--skip-nx-cache` flag.

```console
$ npx lerna run build --skip-nx-cache
```

### Pack

As called out in the above sections, the AWS CDK uses jsii to produce polyglot targets. This means that each CDK module
Expand Down
31 changes: 24 additions & 7 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ runtarget="build"
run_tests="true"
check_prereqs="true"
check_compat="true"
ci="false"
scope=""
while [[ "${1:-}" != "" ]]; do
case $1 in
-h|--help)
Expand All @@ -27,6 +29,9 @@ while [[ "${1:-}" != "" ]]; do
--skip-compat)
check_compat="false"
;;
--ci)
ci=true
;;
*)
echo "Unrecognized parameter: $1"
exit 1
Expand All @@ -48,6 +53,18 @@ fi

echo "============================================================================================="
echo "installing..."
version=$(node -p "require('./package.json').version")
# this is super weird. If you run 'npm install' twice
# and it actually performs an install, then
# node-bundle test will fail with "npm ERR! maxAge must be a number".
# This won't happen in most instances because if nothing changes then npm install
# won't perform an install.
# In the pipeline however, npm install is run once when all the versions are '0.0.0' (via ./scripts/bump-candidate.sh)
# and then `align-versions` is run which updates all the versions to
# (for example) `2.74.0-rc.0` and then npm install is run again here.
if [ "$version" != "0.0.0" ]; then
rm -rf node_modules
fi
yarn install --frozen-lockfile --network-timeout 1000000

fail() {
Expand All @@ -72,21 +89,21 @@ node ./scripts/check-yarn-lock.js
BUILD_INDICATOR=".BUILD_COMPLETED"
rm -rf $BUILD_INDICATOR

# Speed up build by reusing calculated tree hashes
# On dev machine, this speeds up the TypeScript part of the build by ~30%.
export MERKLE_BUILD_CACHE=$(mktemp -d)
trap "rm -rf $MERKLE_BUILD_CACHE" EXIT

if [ "$run_tests" == "true" ]; then
runtarget="$runtarget+test"
runtarget="$runtarget,test"
fi

# Limit top-level concurrency to available CPUs - 1 to limit CPU load.
concurrency=$(node -p 'Math.max(1, require("os").cpus().length - 1)')

flags=""
if [ "$ci" == "true" ]; then
flags="--stream --no-progress --skip-nx-cache"
fi

echo "============================================================================================="
echo "building..."
time npx lerna run $bail --stream --concurrency=$concurrency $runtarget || fail
time npx lerna run $bail --concurrency=$concurrency $runtarget $flags || fail

if [ "$check_compat" == "true" ]; then
/bin/bash scripts/check-api-compatibility.sh
Expand Down
2 changes: 1 addition & 1 deletion buildspec-pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ phases:
- /bin/bash ./scripts/cache-load.sh
build:
commands:
- /bin/bash ./build.sh
- /bin/bash ./build.sh --ci
# After compilation, run Rosetta (using the cache if available).
# This will print errors, and fail the build if there are compilation errors in any packages marked as 'strict'.
- /bin/bash ./scripts/run-rosetta.sh
Expand Down
2 changes: 1 addition & 1 deletion buildspec.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ phases:
- codebuild-breakpoint
- 'if ${BUMP_CANDIDATE:-false}; then env NODE_OPTIONS="--max-old-space-size=8196 ${NODE_OPTIONS:-}" /bin/bash ./scripts/bump-candidate.sh; fi'
- /bin/bash ./scripts/align-version.sh
- /bin/bash ./build.sh
- /bin/bash ./build.sh --ci
post_build:
commands:
# Short-circuit: Don't run pack if the above build failed.
Expand Down
25 changes: 18 additions & 7 deletions lerna.json
Original file line number Diff line number Diff line change
@@ -1,18 +1,29 @@
{
"lerna": "^4.0.0",
"npmClient": "yarn",
"useWorkspaces": true,
"packages": [
"packages/*",
"packages/aws-cdk-lib",
"packages/cdk-cli-wrapper",
"packages/cdk-assets",
"packages/aws-cdk",
"packages/cdk",
"packages/@aws-cdk/*",
"packages/awslint",
"packages/@aws-cdk-containers/*",
"packages/@aws-cdk-testing/*",
"packages/@aws-cdk/*/lambda-packages/*",
"tools/*",
"tools/@aws-cdk/*",
"scripts/@aws-cdk/script-tests",
"packages/individual-packages/*"
"tools/@aws-cdk/cdk-build-tools",
"tools/@aws-cdk/cdk-release",
"tools/@aws-cdk/cfn2ts",
"tools/@aws-cdk/eslint-plugin",
"tools/@aws-cdk/node-bundle",
"tools/@aws-cdk/pkglint",
"tools/@aws-cdk/pkgtools",
"tools/@aws-cdk/prlint",
"tools/@aws-cdk/yarn-cling",
"scripts/@aws-cdk/script-tests"
],
"rejectCycles": "true",
"version": "0.0.0"
"version": "0.0.0",
"$schema": "node_modules/lerna/schemas/lerna-schema.json"
}
60 changes: 60 additions & 0 deletions nx.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
{
"extends": "@nrwl/workspace/presets/npm.json",
"workspaceLayout": { },
"tasksRunnerOptions": {
"default": {
"runner": "@nrwl/workspace/tasks-runners/default",
"options": {
"cacheableOperations": [
"build",
"test",
"lint",
"package",
"prepare"
]
}
}
},
"targetDefaults": {
"build": {
"implicitDependencies": ["aws-cdk-lib"],
"dependsOn": ["^build"],
"inputs": [
"{projectRoot}/lib/!(*.d|*.generated).ts",
"{projectRoot}/test/!(*.d).ts",
"!{workspaceRoot}/**/tsconfig.json",
"!{workspaceRoot}/**/tsconfig.json",
"!{workspaceRoot}/tsconfig.base.json",
"!{workspaceRoot}/**/tsconfig.tsbuildinfo"
],
"outputs": [
"!{projectRoot}/**/*.integ.snapshot/*",
"{projectRoot}/tsconfig.json",
"{projectRoot}/lib/aws-custom-resource/sdk-api-metadata.json",
"{projectRoot}/build-info.json",
"{projectRoot}/lib/**/*.js",
"{projectRoot}/lib/layer.zip",
"{projectRoot}/bin/**/*.js",
"{projectRoot}/lib/**/*.js.map",
"{projectRoot}/lib/**/*.generated.ts",
"{projectRoot}/lib/**/*.d.ts",
"{projectRoot}/test/**/*.d.ts",
"{projectRoot}/test/**/*.js",
"{projectRoot}/.jsii",
"{projectRoot}/spec/*.json",
"{projectRoot}/.warnings.jsii.js"
]
},
"test": {
"dependsOn": ["build"]
},
},
"affected": {
"defaultBase": "main"
},
"pluginsConfig": {
"@nrwl/js": {
"analyzeSourceFiles": false
}
}
}
29 changes: 21 additions & 8 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@
"pack": "./pack.sh",
"compat": "./scripts/check-api-compatibility.sh",
"bump": "./bump.sh",
"build-all": "tsc -b",
"postinstall": "patch-package --error-on-fail"
"build-all": "tsc -b"
},
"devDependencies": {
"@types/node": "18.11.19",
Expand All @@ -28,10 +27,13 @@
"jsii-pacmak": "1.78.1",
"jsii-reflect": "1.78.1",
"jsii-rosetta": "~5.0.0",
"lerna": "^4.0.0",
"lerna": "^6.6.1",
"patch-package": "^6.5.1",
"semver": "^6.3.0",
"standard-version": "^9.5.0",
"@nrwl/cli": "^15.9.1",
"@nrwl/workspace": "^15.9.1",
"nx": "^15.9.1",
"typescript": "~4.9.5"
},
"resolutions": {
Expand Down Expand Up @@ -66,15 +68,26 @@
},
"workspaces": {
"packages": [
"packages/*",
"packages/aws-cdk-lib",
"packages/cdk-cli-wrapper",
"packages/aws-cdk",
"packages/cdk",
"packages/cdk-assets",
"packages/@aws-cdk/*",
"packages/awslint",
"packages/@aws-cdk-containers/*",
"packages/@aws-cdk-testing/*",
"packages/@aws-cdk/*/lambda-packages/*",
"tools/*",
"tools/@aws-cdk/*",
"scripts/@aws-cdk/script-tests",
"packages/individual-packages/*"
"tools/@aws-cdk/cdk-build-tools",
"tools/@aws-cdk/cdk-release",
"tools/@aws-cdk/cfn2ts",
"tools/@aws-cdk/eslint-plugin",
"tools/@aws-cdk/node-bundle",
"tools/@aws-cdk/pkglint",
"tools/@aws-cdk/pkgtools",
"tools/@aws-cdk/prlint",
"tools/@aws-cdk/yarn-cling",
"scripts/@aws-cdk/script-tests"
],
"nohoist": [
"**/jszip",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
{
"version": "21.0.0",
"version": "31.0.0",
"files": {
"37e0c4dfe9c2afeff7fb5beab5ab205de8e42b55f7c46884d976fe647d43c06b": {
"d28780ff347cb790782f733de5ab866ea07bb7ee12ee3e8876c976146f2073d4": {
"source": {
"path": "aws-cdk-vpc-nat-instances.template.json",
"packaging": "file"
},
"destinations": {
"12345678-test-region": {
"bucketName": "cdk-hnb659fds-assets-12345678-test-region",
"objectKey": "37e0c4dfe9c2afeff7fb5beab5ab205de8e42b55f7c46884d976fe647d43c06b.json",
"objectKey": "d28780ff347cb790782f733de5ab866ea07bb7ee12ee3e8876c976146f2073d4.json",
"region": "test-region",
"assumeRoleArn": "arn:${AWS::Partition}:iam::12345678:role/cdk-hnb659fds-file-publishing-role-12345678-test-region"
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1a",
"AvailabilityZone": "dummy1a",
"CidrBlock": "10.0.0.0/19",
"MapPublicIpOnLaunch": true,
"Tags": [
Expand Down Expand Up @@ -93,7 +93,7 @@
"MyVpcPublicSubnet1NatInstance8E94E5F7": {
"Type": "AWS::EC2::Instance",
"Properties": {
"AvailabilityZone": "test-region-1a",
"AvailabilityZone": "dummy1a",
"IamInstanceProfile": {
"Ref": "MyVpcPublicSubnet1NatInstanceInstanceProfile2FD934CB"
},
Expand Down Expand Up @@ -131,7 +131,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1b",
"AvailabilityZone": "dummy1b",
"CidrBlock": "10.0.32.0/19",
"MapPublicIpOnLaunch": true,
"Tags": [
Expand Down Expand Up @@ -203,7 +203,7 @@
"MyVpcPublicSubnet2NatInstance04BCE4E3": {
"Type": "AWS::EC2::Instance",
"Properties": {
"AvailabilityZone": "test-region-1b",
"AvailabilityZone": "dummy1b",
"IamInstanceProfile": {
"Ref": "MyVpcPublicSubnet2NatInstanceInstanceProfile5AB09EF6"
},
Expand Down Expand Up @@ -241,7 +241,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1c",
"AvailabilityZone": "dummy1c",
"CidrBlock": "10.0.64.0/19",
"MapPublicIpOnLaunch": true,
"Tags": [
Expand Down Expand Up @@ -306,7 +306,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1a",
"AvailabilityZone": "dummy1a",
"CidrBlock": "10.0.96.0/19",
"MapPublicIpOnLaunch": false,
"Tags": [
Expand Down Expand Up @@ -368,7 +368,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1b",
"AvailabilityZone": "dummy1b",
"CidrBlock": "10.0.128.0/19",
"MapPublicIpOnLaunch": false,
"Tags": [
Expand Down Expand Up @@ -430,7 +430,7 @@
"VpcId": {
"Ref": "MyVpcF9F0CA6F"
},
"AvailabilityZone": "test-region-1c",
"AvailabilityZone": "dummy1c",
"CidrBlock": "10.0.160.0/19",
"MapPublicIpOnLaunch": false,
"Tags": [
Expand Down
Original file line number Diff line number Diff line change
@@ -1 +1 @@
{"version":"20.0.0"}
{"version":"31.0.0"}
Original file line number Diff line number Diff line change
@@ -1,14 +1,12 @@
{
"version": "21.0.0",
"version": "31.0.0",
"testCases": {
"integ.nat-instances.lit": {
"integ-test/DefaultTest": {
"stacks": [
"aws-cdk-vpc-nat-instances"
],
"diffAssets": false,
"stackUpdateWorkflow": true
"assertionStack": "integ-test/DefaultTest/DeployAssert",
"assertionStackName": "integtestDefaultTestDeployAssert24D5C536"
}
},
"synthContext": {},
"enableLookups": true
}
}
Loading

0 comments on commit 046d0fc

Please sign in to comment.