Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci3 #10775

Open
22 of 41 tasks
ludamad opened this issue Dec 16, 2024 · 0 comments
Open
22 of 41 tasks

ci3 #10775

ludamad opened this issue Dec 16, 2024 · 0 comments
Assignees
@ludamad @charlielye

Comments

@ludamad
Copy link
Collaborator

ludamad commented Dec 16, 2024 •
edited by charlielye
Loading

Full CI3:

  • Get rid of barretenberg/cpp/srs_db. Modify code to assume dat is in ~/.bb-crs (in flat format i.e. single file, no header). Will save 50s on build and prob some $.
  • Handle spot termination with immediate on-demand retry.
  • Add loud warnings/alerts around unexpectedly long build times. Both arches. e.g. currently we have a painful 10-20m build issue on arm with avm relations/poseidon2.cpp.
  • We may need to expand instance scope when requesting instances. Currently just m6a.32xlarge. But I'd like to see how we get on first. However need to prepare to respond if issues. Probably just loop instance by instance in order of preference. Slow but simple. It's an edge case.
  • Explore zero-install or caching node modules to avoid slow yarn installs.
  • All formatting/linting should be part of build, not test. Thus once built and cached, we don't do it again.
  • Real-time regenerate inputs for theverify_honk_proof acir tests.
  • Make ci image actually just be the dev container image, with all dev tools e.g. zsh, htop etc.
  • No building of build-image in CI. Just have a ./bootstrap.sh deploy that will build and push it. Can be done from mainframe.
  • Figure out how to rebuild and change the build-instance ami in aws to ensure we don't always need to pull image (maybe too much effort, but need to consider the data transfter costs. ECR?).
  • Bake CRS into AMI and mount it into container so we don't have to download about 2GB...
  • Docs build.
  • BB gcc build? Or on masterly? Can we use -fsyntax-only
  • Improve our rebuild patterns. Support e.g. inversions !some/script/that/should/not/trigger
  • Enable "merge queues" and have it follow a different "masterly" workflow.
  • Create the final slim release container, from the s3 cache artefacts, on masterly.
  • Run benchmarks on masterly.
  • Deploy if the merge was a release.
  • Rebuild sysbox to ensure it reflects toolchain changes (e.g. move to global yarn 4).
  • Make obscure errors more explicit (e.g. jq error on vk key gen)
  • Delete every Dockerfile, Earthfile, bash script, and Yaml file, that isn't needed, or can be replaced with something simpler.
  • Make a failing fast with no local changes print large announcement to alert ci team.
  • Contact tests and aztec-nr tests (txe means yarn-project circular).
  • Fix commented noir js tests (bb circular).
  • Make bb/.js crs download only download whats needed to extend an existing CRS. Will make the process idempotent and then we don't need to worry about process races.
  • test-all: Skipping tests matching test_caches_open|requests in noir tests.
  • test-all: kv-store excluded cos mocha.
  • test-all: noir js packages tests?
  • test-all: Standardise how tests are skipped so we can see in one place.
  • Maintain a fleet of spots that will live for up to e.g. 1 hour idle. Any spots request will come from the fleet if available. Will give faster startup times.
  • Put time -v / ulimit on tests and alert/kill on mem usage.
  • TXE should exit with 0 on SIGTERM.
  • Better secret scoping. e.g. only tagged releases should have access to dockerhub key. Tags can only be created by authorized flow (i.e. at least a PR with an approval).
  • Handle redis connection failures gracefully. e.g. at present the test filter, if fails to find redis, just runs no tests.

Disabled flakes:
@ludamad ludamad mentioned this issue Dec 17, 2024
Merged
charlielye added a commit that referenced this issue Dec 17, 2024
@ludamad @MirandaWood
@charlielye @TomAFrench
chore(ci): towards ci3 (#10751)
dee6ba3 
CI3 is a conceptual goal for uniting the CI flow and the dev flow as
much as possible, adding more depth to the bootstrap and build scripts
to be able to handle our needs.

This PR introduces all the work on CI3 so far, but still has an earthly
caller shell to make sure we can minimize the number of variables that
have changed at once.

There is a lot of changes in this PR.
See https://github.com/AztecProtocol/aztec-packages/pull/10711/files for
a subset of the changes without yarn.lock etc noise.

The big picture:
- The CI build has been made much less stateful. ci.yml now uses the ci3
bootstrap pattern, without fully moving off the earthly targets just
yet.
- The S3 cache mechanism is now the main cache mechanism. Note there is
no persistent disk now supporting the build.
There is a global cache on S3, readable without auth, that caches them
for 10 days. We no longer think of the build in terms of docker/buildkit
layers but instead as chunks that have different rebuild patterns that
match files in the monorepo.
- Moving to yarn 4.5.2. 

Niceties:
- faster builds due to script improvements and distributed cache
uploading by default
- work is more properly isolated in chunks from the above effort
- spot recovery is implemented, retrying with on-demand
- we no longer use github runners, side-stepping lots of edge-cases, and
instead rely on our builder realizing there is no work to do / hitting a
timeout via shutdown -P
- Docker images are no longer copied from the builder, meaning a large
class of flake is gone.

Non-niceties:
- The earthly setup is much less granular. There is two stages that have
their own one-layer builds. The earthly cache is fairly redundant, using
the S3 cache for most meaningful caching. (earthly will not be used in
ci.yml in the future)
- Some CI files are now duplicated, we will do a follow-on pass to get
rid of earthly helpers, build-system, etc
- CI currently also downloads the CI image fresh each time, will change
- we are currently pushing images to dockerhub with no expiration,
should move to ECR
- noir-projects currently retries once in the Earthfile as a last minute
issue was hit, will be fixed in a follow-up
- Issues: #10775

---- 
WORKFLOW AFTER THIS PR:
- Run ./bootstrap.sh in root to bootstrap with cache, ./bootstrap.sh
full otherwise
- Run earthly +ci in root to 
- Put ci3 in your cache and note the ways to interact with ci in that
folder
- Note the new commands in ./bootstrap.sh like test-kind-network
- In yarn-project to run a single e2e test now use `test:e2e`. `test`
just runs the unit tests as per other projects.

---------

Co-authored-by: MirandaWood <[email protected]>
Co-authored-by: Charlie Lye <[email protected]>
Co-authored-by: Tom French <[email protected]>
@charlielye charlielye mentioned this issue Dec 17, 2024
Merged
ludamad added a commit that referenced this issue Dec 19, 2024
@charlielye @ludamad
@MirandaWood @TomAFrench
chore: ci3 phase 1 (#10042)
641da4b 
CI3 is a conceptual goal for uniting the CI flow and the dev flow as
much as possible, adding more depth to the bootstrap and build scripts
to be able to handle our needs.

This PR introduces all the work on CI3 so far, but still has an earthly
caller shell to make sure we can minimize the number of variables that
have changed at once.

There is a lot of changes in this PR.
See https://github.com/AztecProtocol/aztec-packages/pull/10711/files for
a subset of the changes without yarn.lock etc noise.

The big picture:
- The CI build has been made much less stateful. ci.yml now uses the ci3
bootstrap pattern, without fully moving off the earthly targets just
yet.
- The S3 cache mechanism is now the main cache mechanism. Note there is
no persistent disk now supporting the build.
There is a global cache on S3, readable without auth, that caches them
for 10 days. We no longer think of the build in terms of docker/buildkit
layers but instead as chunks that have different rebuild patterns that
match files in the monorepo.
- Moving to yarn 4.5.2. 

Niceties:
- faster builds due to script improvements and distributed cache
uploading by default
- work is more properly isolated in chunks from the above effort
- spot recovery is implemented, retrying with on-demand
- we no longer use github runners, side-stepping lots of edge-cases, and
instead rely on our builder realizing there is no work to do / hitting a
timeout via shutdown -P
- Docker images are no longer copied from the builder, meaning a large
class of flake is gone.

Non-niceties:
- The earthly setup is much less granular. There is two stages that have
their own one-layer builds. The earthly cache is fairly redundant, using
the S3 cache for most meaningful caching. (earthly will not be used in
ci.yml in the future)
- Some CI files are now duplicated, we will do a follow-on pass to get
rid of earthly helpers, build-system, etc
- CI currently also downloads the CI image fresh each time, will change
- we are currently pushing images to dockerhub with no expiration,
should move to ECR
- noir-projects currently retries once in the Earthfile as a last minute
issue was hit, will be fixed in a follow-up
- Issues: #10775

---- 
WORKFLOW AFTER THIS PR:
- Run ./bootstrap.sh in root to bootstrap using the (publicly available)
S3 cache, ./bootstrap.sh
full to force a full build
- Run ./bootstrap.sh ci to test the in-progress 'full CI3' locally
- Run ./ci.sh ec2 to test the in-progress 'full CI3' on an isolated
runner
- For ci2.5, use earthly +ci in root to simulate ci.yml. This shares the
S3 cache. Make sure to alias earthly to scripts/earthly_local.
- Put ci3 in your cache and note the ways to interact with ci in that
folder
- Recommended workflow, as commits are now needed to run earthly or
bootstrap_ec2:
```
# in repo root
./ci.sh draft && git commit -am "blobs work" && git push && earthly +ci
```
Other useful ci.sh commands are gha-url to see the last github job
associated with your branch.
- Notable commands in ./bootstrap.sh are test-kind-network, test-e2e,
images-e2e
- In yarn-project to run a single e2e test now use `test:e2e`. `test`
just runs the unit tests as per other projects.

---------

Co-authored-by: ludamad <[email protected]>
Co-authored-by: MirandaWood <[email protected]>
Co-authored-by: Tom French <[email protected]>
Co-authored-by: ludamad <[email protected]>
AztecBot pushed a commit to AztecProtocol/barretenberg that referenced this issue Dec 20, 2024
@charlielye @ludamad
@MirandaWood @TomAFrench @AztecBot
chore: ci3 phase 1 (#10042)
a8cf840 
CI3 is a conceptual goal for uniting the CI flow and the dev flow as
much as possible, adding more depth to the bootstrap and build scripts
to be able to handle our needs.

This PR introduces all the work on CI3 so far, but still has an earthly
caller shell to make sure we can minimize the number of variables that
have changed at once.

There is a lot of changes in this PR.
See https://github.com/AztecProtocol/aztec-packages/pull/10711/files for
a subset of the changes without yarn.lock etc noise.

The big picture:
- The CI build has been made much less stateful. ci.yml now uses the ci3
bootstrap pattern, without fully moving off the earthly targets just
yet.
- The S3 cache mechanism is now the main cache mechanism. Note there is
no persistent disk now supporting the build.
There is a global cache on S3, readable without auth, that caches them
for 10 days. We no longer think of the build in terms of docker/buildkit
layers but instead as chunks that have different rebuild patterns that
match files in the monorepo.
- Moving to yarn 4.5.2. 

Niceties:
- faster builds due to script improvements and distributed cache
uploading by default
- work is more properly isolated in chunks from the above effort
- spot recovery is implemented, retrying with on-demand
- we no longer use github runners, side-stepping lots of edge-cases, and
instead rely on our builder realizing there is no work to do / hitting a
timeout via shutdown -P
- Docker images are no longer copied from the builder, meaning a large
class of flake is gone.

Non-niceties:
- The earthly setup is much less granular. There is two stages that have
their own one-layer builds. The earthly cache is fairly redundant, using
the S3 cache for most meaningful caching. (earthly will not be used in
ci.yml in the future)
- Some CI files are now duplicated, we will do a follow-on pass to get
rid of earthly helpers, build-system, etc
- CI currently also downloads the CI image fresh each time, will change
- we are currently pushing images to dockerhub with no expiration,
should move to ECR
- noir-projects currently retries once in the Earthfile as a last minute
issue was hit, will be fixed in a follow-up
- Issues: AztecProtocol/aztec-packages#10775

---- 
WORKFLOW AFTER THIS PR:
- Run ./bootstrap.sh in root to bootstrap using the (publicly available)
S3 cache, ./bootstrap.sh
full to force a full build
- Run ./bootstrap.sh ci to test the in-progress 'full CI3' locally
- Run ./ci.sh ec2 to test the in-progress 'full CI3' on an isolated
runner
- For ci2.5, use earthly +ci in root to simulate ci.yml. This shares the
S3 cache. Make sure to alias earthly to scripts/earthly_local.
- Put ci3 in your cache and note the ways to interact with ci in that
folder
- Recommended workflow, as commits are now needed to run earthly or
bootstrap_ec2:
```
# in repo root
./ci.sh draft && git commit -am "blobs work" && git push && earthly +ci
```
Other useful ci.sh commands are gha-url to see the last github job
associated with your branch.
- Notable commands in ./bootstrap.sh are test-kind-network, test-e2e,
images-e2e
- In yarn-project to run a single e2e test now use `test:e2e`. `test`
just runs the unit tests as per other projects.

---------

Co-authored-by: ludamad <[email protected]>
Co-authored-by: MirandaWood <[email protected]>
Co-authored-by: Tom French <[email protected]>
Co-authored-by: ludamad <[email protected]>
@charlielye charlielye mentioned this issue Jan 3, 2025
Merged
charlielye added a commit that referenced this issue Feb 14, 2025
@charlielye @ludamad @Thunkar
feat: Cl/ci3.3 (#10946)
723163a 
[CI3 introduction.](https://hackmd.io/bTnKHtTHT8mAdTtD0t7JvA?view)

This is a majority step towards the vision of CI3, still namely missing
merge queue.

New features:
- Grinding flakes in master. We run all tests on 5 separate runners to
report on flakes at the source.
- External contributors can now have CI run just by approving their PR. 
- Ability to debug CI entirely from commandline from any machine. Get
dropped into a productive shell right after the CI failure by doing
`./ci.sh ec2` while your PR is a draft (note: do not do this if pushing
to a non-draft PR).
- Add tests to CI by adding tests to bootstrap. Target a rich
environment with no differences from running inside the dev container.
- Releases that are fully dry-runnable and deployable from a single
command. See above hackmd for details.
- Recovery from spot eviction (finally implemented correctly).

Some remaining items are tracked here.
#10775

---------

Co-authored-by: ludamad <[email protected]>
Co-authored-by: ludamad <[email protected]>
Co-authored-by: thunkar <[email protected]>
AztecBot pushed a commit to AztecProtocol/barretenberg that referenced this issue Feb 15, 2025
@charlielye @ludamad
@Thunkar @AztecBot
feat: Cl/ci3.3 (#10946)
d0161bf 
[CI3 introduction.](https://hackmd.io/bTnKHtTHT8mAdTtD0t7JvA?view)

This is a majority step towards the vision of CI3, still namely missing
merge queue.

New features:
- Grinding flakes in master. We run all tests on 5 separate runners to
report on flakes at the source.
- External contributors can now have CI run just by approving their PR. 
- Ability to debug CI entirely from commandline from any machine. Get
dropped into a productive shell right after the CI failure by doing
`./ci.sh ec2` while your PR is a draft (note: do not do this if pushing
to a non-draft PR).
- Add tests to CI by adding tests to bootstrap. Target a rich
environment with no differences from running inside the dev container.
- Releases that are fully dry-runnable and deployable from a single
command. See above hackmd for details.
- Recovery from spot eviction (finally implemented correctly).

Some remaining items are tracked here.
AztecProtocol/aztec-packages#10775

---------

Co-authored-by: ludamad <[email protected]>
Co-authored-by: ludamad <[email protected]>
Co-authored-by: thunkar <[email protected]>
AztecBot pushed a commit to AztecProtocol/aztec-nr that referenced this issue Feb 15, 2025
@charlielye @ludamad
@Thunkar @AztecBot
feat: Cl/ci3.3 (#10946)
19b8dce 
[CI3 introduction.](https://hackmd.io/bTnKHtTHT8mAdTtD0t7JvA?view)

This is a majority step towards the vision of CI3, still namely missing
merge queue.

New features:
- Grinding flakes in master. We run all tests on 5 separate runners to
report on flakes at the source.
- External contributors can now have CI run just by approving their PR. 
- Ability to debug CI entirely from commandline from any machine. Get
dropped into a productive shell right after the CI failure by doing
`./ci.sh ec2` while your PR is a draft (note: do not do this if pushing
to a non-draft PR).
- Add tests to CI by adding tests to bootstrap. Target a rich
environment with no differences from running inside the dev container.
- Releases that are fully dry-runnable and deployable from a single
command. See above hackmd for details.
- Recovery from spot eviction (finally implemented correctly).

Some remaining items are tracked here.
AztecProtocol/aztec-packages#10775

---------

Co-authored-by: ludamad <[email protected]>
Co-authored-by: ludamad <[email protected]>
Co-authored-by: thunkar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ludamad ludamad

@charlielye charlielye

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ludamad @charlielye