Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: use s3 cache in bootstrap fast #9111

Merged
merged 20 commits into from
Oct 11, 2024
Merged

feat: use s3 cache in bootstrap fast #9111

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
a4e9214
feat: use s3 cache in bootstrap fast
alexghr Oct 9, 2024
5955d7b
fix: bb.js caching
alexghr Oct 9, 2024
7897420
fix: rebuild patterns
alexghr Oct 9, 2024
d9e6864
fix: cache noir projects
alexghr Oct 10, 2024
5f9391c
refactor: remove reliance on setup_env
alexghr Oct 10, 2024
36b7dbc
fix: remove aws creds from BASH_ENV
alexghr Oct 10, 2024
7cb9e9c
fix: variants
alexghr Oct 10, 2024
a774265
Update cache-download.sh
ludamad Oct 10, 2024
d002b8f
fix: save whole l1-contracts folder as artifact
alexghr Oct 10, 2024
041461f
fix: more fixes
alexghr Oct 10, 2024
da8e1fd
fix: cache scripts
alexghr Oct 10, 2024
5aa2882
chore: revert dimensions
alexghr Oct 10, 2024
980d6f3
fix: noir proto circuits
alexghr Oct 10, 2024
cc5d511
fix: don't exit
alexghr Oct 10, 2024
0c72b5c
chore: bump the cache
alexghr Oct 10, 2024
516cb91
fix: cache autogenerated
alexghr Oct 10, 2024
3f297e7
fix: bootstrap_cache scripts
alexghr Oct 10, 2024
71c005a
fix: parallel download
alexghr Oct 11, 2024
d887bbe
fix: cache
alexghr Oct 11, 2024
b25a3ed
Merge branch 'master' into ag/cache
alexghr Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions barretenberg/cpp/bootstrap_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,16 @@ function on_exit() {
}
trap on_exit EXIT

$SCRIPTS_PATH/cache-download.sh barretenberg-preset-wasm-$HASH.tar.gz $TMP/build-wasm
$SCRIPTS_PATH/cache-download.sh barretenberg-preset-wasm-threads-$HASH.tar.gz $TMP/build-wasm-threads
$SCRIPTS_PATH/cache-download.sh barretenberg-preset-release-$HASH.tar.gz $TMP/build-release
$SCRIPTS_PATH/cache-download.sh barretenberg-preset-release-world-state-$HASH.tar.gz $TMP/build-world-state
# Parallel download of all the cached builds because they're quite big
echo "
barretenberg-preset-wasm
barretenberg-preset-wasm-threads
barretenberg-preset-release
barretenberg-preset-release-world-state
" | xargs --max-procs 0 -I {} bash -c "$SCRIPTS_PATH/cache-download.sh {}-$HASH.tar.gz $TMP/{}"

# clobber the existing build with the cached build
mv -f $TMP/build-wasm/build build-wasm/
mv -f $TMP/build-wasm-threads/build build-wasm-threads/
mv -f $TMP/build-release/build build/
mv -f $TMP/build-world-state/build/bin/* build/bin/
# # clobber the existing build with the cached build
cp -r $TMP/barretenberg-preset-wasm/build build-wasm/
cp -r $TMP/barretenberg-preset-wasm-threads/build build-wasm-threads/
cp -r $TMP/barretenberg-preset-release/build build/
cp -r $TMP/barretenberg-preset-release-world-state/build/* build/
17 changes: 10 additions & 7 deletions barretenberg/ts/bootstrap_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,16 @@ trap on_exit EXIT

HASH=$(AZTEC_CACHE_REBUILD_PATTERNS="../cpp/.rebuild_patterns .rebuild_patterns" $CACHE_SCRIPTS/compute-content-hash.sh)

$CACHE_SCRIPTS/cache-download.sh bb.js-esm-$HASH.tar.gz $TMP/esm
$CACHE_SCRIPTS/cache-download.sh bb.js-cjs-$HASH.tar.gz $TMP/cjs
$CACHE_SCRIPTS/cache-download.sh bb.js-browser-$HASH.tar.gz $TMP/browser

cp -r $TMP/esm/dest dest
cp -r $TMP/cjs/dest dest
cp -r $TMP/browser/dest dest
# Parallel download of all the cached builds because they're quite big
echo "
bb.js-esm
bb.js-cjs
bb.js-browser
" | xargs --max-procs 0 -I {} bash -c "$SCRIPTS_PATH/cache-download.sh {}-$HASH.tar.gz $TMP/{}"

cp -r $TMP/bb.js-esm/dest dest
cp -r $TMP/bb.js-cjs/dest dest
cp -r $TMP/bb.js-browser/dest dest

# Annoyingly we still need to install modules, so they can be found as part of module resolution when portalled.
yarn install
2 changes: 1 addition & 1 deletion noir-projects/Earthfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ build-protocol-circuits:
WORKDIR /usr/src/noir-projects/noir-protocol-circuits

DO ../build-system/s3-cache-scripts/+WITH_CACHE \
--prefix="noir-projects-noir-protocol-circuits-2" \
--prefix="noir-projects-noir-protocol-circuits" \
--rebuild_patterns="../../noir/.rebuild_patterns_native ../../barretenberg/cpp/.rebuild_patterns .rebuild_patterns" \
--command="./bootstrap.sh" \
--build_artifacts="target Nargo.toml private_kernel_reset_dimensions.json crates/autogenerated"
Expand Down
13 changes: 6 additions & 7 deletions noir-projects/bootstrap_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,13 @@ SCRIPTS_PATH=../build-system/s3-cache-scripts/
echo -e "\033[1mRetrieving noir projects from remote cache...\033[0m"

PROTOCOL_CIRCUITS_HASH=$(AZTEC_CACHE_REBUILD_PATTERNS="../noir/.rebuild_patterns_native ../barretenberg/cpp/.rebuild_patterns ./noir-protocol-circuits/.rebuild_patterns" $SCRIPTS_PATH/compute-content-hash.sh)
$SCRIPTS_PATH/cache-download.sh noir-projects-noir-protocol-circuits-2-$PROTOCOL_CIRCUITS_HASH.tar.gz noir-protocol-circuits

MOCK_CIRCUITS_HASH=$(AZTEC_CACHE_REBUILD_PATTERNS="../noir/.rebuild_patterns_native ../barretenberg/cpp/.rebuild_patterns ./mock-protocol-circuits/.rebuild_patterns" $SCRIPTS_PATH/compute-content-hash.sh)
$SCRIPTS_PATH/cache-download.sh noir-projects-mock-protocol-circuits-$MOCK_CIRCUITS_HASH.tar.gz mock-protocol-circuits

CONTRACTS_HASH=$(AZTEC_CACHE_REBUILD_PATTERNS="../noir/.rebuild_patterns_native ../avm-transpiler/.rebuild_patterns ../barretenberg/cpp/.rebuild_patterns noir-contracts/.rebuild_patterns" $SCRIPTS_PATH/compute-content-hash.sh)
$SCRIPTS_PATH/cache-download.sh noir-projects-noir-contracts-$CONTRACTS_HASH.tar.gz noir-contracts

yarn
echo "
noir-protocol-circuits
mock-protocol-circuits
noir-contracts
" | xargs --max-procs 0 -I {} bash -c "$SCRIPTS_PATH/cache-download.sh noir-projects-{}-$HASH.tar.gz {}"

(cd ./noir-protocol-circuits && yarn && node ./scripts/generate_variants.js)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is captured in cache now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the script creates the crates/autogenerated directory, the private_kernel_reset_dimensions.json file and root Nargo.toml. All three are part of the artifacts archive now :D

yarn
2 changes: 0 additions & 2 deletions noir-projects/noir-protocol-circuits/bootstrap.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,4 @@ else
for pathname in "./target"/*.json; do
BB_HASH=$BB_HASH node ../scripts/generate_vk_json.js "$pathname" "./target/keys"
done


fi
Loading