Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: improve dependency installation in rust builder #232

Merged
merged 36 commits into from
May 13, 2024
Merged

fix: improve dependency installation in rust builder #232

merged 36 commits into from
May 13, 2024

Conversation

bkioshn
Copy link
Contributor

@bkioshn bkioshn commented Apr 15, 2024 •
edited
Loading

Description

Improve rust related performance

Related Issue(s)

Closes #228

Description of Changes

  • Use lib/rust https://github.com/earthly/lib/tree/main/rust
    • Additionally, FUNCTION EXECUTE, CARGO, and COPY_OUTPUT are modified and added to suit our usecase
  • Use IMPORT instead of path when refering other Earthfile
  • Rust tools parallel building
  • Modifying the Rust document

Result of this changes:

  • Changing of one tool doesn't effect the other tools to be rebuilt
  • Efficient use of cache result in faster workflow
  • Cleaner code implementation

Related PR

Please confirm the following checks

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream module

@bkioshn bkioshn marked this pull request as draft April 15, 2024 06:44
@bkioshn bkioshn self-assigned this Apr 15, 2024
@bkioshn bkioshn added enhancement New feature or request comment me Request for comments labels Apr 15, 2024
bkioshn and others added 9 commits April 19, 2024 07:22
@bkioshn
fix: remove name
1b2717e
@bkioshn
Merge branch 'master' into fix/improve-rust-builder
33761f5
@bkioshn
feat: add wasm-pack
50f05c1
@bkioshn
fix: typo
22177f7
@bkioshn
test: try lib/rust earthly
fc264ba
@bkioshn
fix: typo
6b58d94
@bkioshn
fix: remove artifact name
68ff8bb
@stevenj
fix(rust-builders): Improve rust builder (#236)
f6269c6 
* fix(cat-ci): Vendor file check crashes when files are not found, make it robust against this.

* fix(cat-ci): Improve and add tests for using upstream rust build library

* fix(rust): Need real find for earthly rust lib, plus handle saving test/coverage build artifacts on failure.

* Don't commit test/coverage artifacts into the repo.

* Remove test code and obsoleted TRY/FINALLY

* fix(cspell): spelling corrections

* fix(rust): Don't need to build the example verbosely

* fix(rust): fix dbviz builder to use the new cached rust builder

* fix(rust): Remove unused imports from Earthfiles

* fix(rust): Only use imports to define import paths, not directly

* fix(rust): Add shim so consumers of cat-ci are always aligned with our upstream dependency.

* fix(rust): save doc and release artifacts from rust builds

* fix(rust): Do not need top escape the backslash

* fix(rust): Fix output target regex

* fix(rust): Document that rust upstream library should not be used directly, and make tools conformant.

* fix(cspell): spelling correction

* fix(cat-ci): Remove direct Earthfile references and use IMPORT consistently

* fix(rust): --output must refer to files not directories

* fix(cat-ci): misspelled directory

* fix(rust): linting was not done in release, so it had to rebuild everything in debug mode

* fix(rust): Properly copy artifacts from builds and ensure we don't accidentally make debug target builds

* fix(cat-ci): Fix wasm-c reference

* fix(rust): Make it easier to save docs artifacts

* fix(cat-ci): Earthly is case sensitive

* feat(rust): Update the verify-component-adapter tool to v20.0.0
@bkioshn
fix: update docs
62e2c54
@bkioshn bkioshn added documentation Improvements or additions to documentation ci/cd CI/CD Fixes or Improvements. review me PR is ready for review and removed comment me Request for comments labels May 5, 2024
@stevenj
feat(rust): Add parallel building of Tooling and Rust code (#240)
aec6a22 
* fix(cat-ci): Vendor file check crashes when files are not found, make it robust against this.

* fix(cat-ci): Improve and add tests for using upstream rust build library

* fix(rust): Need real find for earthly rust lib, plus handle saving test/coverage build artifacts on failure.

* Don't commit test/coverage artifacts into the repo.

* Remove test code and obsoleted TRY/FINALLY

* fix(cspell): spelling corrections

* fix(rust): Don't need to build the example verbosely

* fix(rust): fix dbviz builder to use the new cached rust builder

* fix(rust): Remove unused imports from Earthfiles

* fix(rust): Only use imports to define import paths, not directly

* fix(rust): Add shim so consumers of cat-ci are always aligned with our upstream dependency.

* fix(rust): save doc and release artifacts from rust builds

* fix(rust): Do not need top escape the backslash

* fix(rust): Fix output target regex

* fix(rust): Document that rust upstream library should not be used directly, and make tools conformant.

* fix(cspell): spelling correction

* fix(cat-ci): Remove direct Earthfile references and use IMPORT consistently

* fix(rust): --output must refer to files not directories

* fix(cat-ci): misspelled directory

* fix(rust): linting was not done in release, so it had to rebuild everything in debug mode

* fix(rust): Properly copy artifacts from builds and ensure we don't accidentally make debug target builds

* fix(cat-ci): Fix wasm-c reference

* fix(rust): Make it easier to save docs artifacts

* fix(cat-ci): Earthly is case sensitive

* feat(rust): Update the verify-component-adapter tool to v20.0.0

* fix(rust): Rust tools build in parallel, no need to use +CARGO to build tools.  Slows build process.

* feat(rust): Add CPU parallel runner for the exec manager

* feat(rust): Use the CPU parallel runner to speed up rust builds on multicore machines

* fix(rust): spelling correction
@stevenj stevenj marked this pull request as ready for review May 6, 2024 02:50
@minikin minikin enabled auto-merge (squash) May 6, 2024 06:15
minikin
minikin requested changes May 6, 2024
View reviewed changes
docs/src/guides/languages/rust.md Outdated Show resolved Hide resolved
minikin
minikin previously requested changes May 6, 2024
View reviewed changes
earthly/rust/stdcfgs/rust-toolchain.toml Outdated Show resolved Hide resolved
@stevenj
fix(rust): Fix/improve rust builder (#243)
54a0251 
* fix(cat-ci): Vendor file check crashes when files are not found, make it robust against this.

* fix(cat-ci): Improve and add tests for using upstream rust build library

* fix(rust): Need real find for earthly rust lib, plus handle saving test/coverage build artifacts on failure.

* Don't commit test/coverage artifacts into the repo.

* Remove test code and obsoleted TRY/FINALLY

* fix(cspell): spelling corrections

* fix(rust): Don't need to build the example verbosely

* fix(rust): fix dbviz builder to use the new cached rust builder

* fix(rust): Remove unused imports from Earthfiles

* fix(rust): Only use imports to define import paths, not directly

* fix(rust): Add shim so consumers of cat-ci are always aligned with our upstream dependency.

* fix(rust): save doc and release artifacts from rust builds

* fix(rust): Do not need top escape the backslash

* fix(rust): Fix output target regex

* fix(rust): Document that rust upstream library should not be used directly, and make tools conformant.

* fix(cspell): spelling correction

* fix(cat-ci): Remove direct Earthfile references and use IMPORT consistently

* fix(rust): --output must refer to files not directories

* fix(cat-ci): misspelled directory

* fix(rust): linting was not done in release, so it had to rebuild everything in debug mode

* fix(rust): Properly copy artifacts from builds and ensure we don't accidentally make debug target builds

* fix(cat-ci): Fix wasm-c reference

* fix(rust): Make it easier to save docs artifacts

* fix(cat-ci): Earthly is case sensitive

* feat(rust): Update the verify-component-adapter tool to v20.0.0

* fix(rust): Rust tools build in parallel, no need to use +CARGO to build tools.  Slows build process.

* feat(rust): Add CPU parallel runner for the exec manager

* feat(rust): Use the CPU parallel runner to speed up rust builds on multicore machines

* fix(rust): spelling correction

* feat(rust): Add WASM toolchains to the base rust builder image.

* fix(rust): Make copy less repetitive and add cargo-component to replace the component adapter

* fix(rust): Update to 1.78, remove unused verify-adapter tool, start using the wasm32-wasip1 target

* fix(rust): try and stop cargo from trying to install

* fix(rust): Dont change the RUST paths from what the base container sets them to

* fix(rust): Try and find why our toolchain keeps getting re-installed

* fix(rust): Don't copy the toolchain as its forcing rustup to redownload the toolchain

* fix(rust): Remove test statements and fully remove rust-toolchain from CI.

* docs(rust): Fix up target docs

* fix(rust): Don't check rust toolchain is vendored.

* fix(rust): make libs and bins in std_build just need comma, not comma and space

* fix(c): Fix tool import from the rust tools

* fix(rust): debug code for argument parsing

* fix(rust): more parameter tests for rust builder

* fix(rust): try and fix rust builder escaping because of parameter munging

* fix(rust): more python arg testing

* fix(rust): more python parameter debugging

* fix(rust): more param debug

* fix(rust): more argument debug

* fix(rust): strip quotes after correction

* fix(rust): actually save the dequoted string.

* fix(rust): remove argument correction debug code

* fix(python): Add reusable utility function to fix quoted earthly arguments

* fix(rust): Allow multiple arguments to +EXECUTE

* fix(rust): missing line continuation

* fix(rust): allow redirection by defering the return fail check

* fix(rust): clean up +EXECUTE

* fix(rust): Remove weird lines that appeared in the file

* docs(rust): fix spelling

* docs(rust): Fill in blank doc comments

* fix(rust): update deny.toml to latest format

* fix(rust): Only warn on crates with multiple versions because we can;t control what our dependencies depend on.

* fix(rust): Ignore unmaintained crate we have no control over.

* fix(rust): Sync the deny.toml from stdcfgs

* fix(cspell): Fix spelling errors in cargo deny.toml

* fix(docs): Save doc on example build as an Artifact and also add target docs

* ci(rust): Add exception to deny.toml for a CVE which is not impacting us.
@stevenj stevenj dismissed minikin’s stale review May 10, 2024 14:17

Change made as requested

@stevenj stevenj self-assigned this May 10, 2024
Mr-Leshiy
Mr-Leshiy reviewed May 12, 2024
View reviewed changes
.config/dictionaries/project.dic Outdated Show resolved Hide resolved
earthly/rust/Earthfile Show resolved Hide resolved
utilities/dbviz/Earthfile Outdated Show resolved Hide resolved
Mr-Leshiy
Mr-Leshiy approved these changes May 13, 2024
View reviewed changes
Copy link
Contributor

@Mr-Leshiy Mr-Leshiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@minikin minikin merged commit 8538908 into master May 13, 2024
60 checks passed
@minikin minikin deleted the fix/improve-rust-builder branch May 13, 2024 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@apskhem apskhem apskhem left review comments

@stevenj stevenj stevenj left review comments

@Mr-Leshiy Mr-Leshiy Mr-Leshiy approved these changes

@minikin minikin minikin left review comments

@saibatizoku saibatizoku Awaiting requested review from saibatizoku

@coire1 coire1 Awaiting requested review from coire1

@FelipeRosa FelipeRosa Awaiting requested review from FelipeRosa

@cong-or cong-or Awaiting requested review from cong-or

Assignees

@stevenj stevenj

@bkioshn bkioshn

Labels
ci/cd CI/CD Fixes or Improvements. documentation Improvements or additions to documentation enhancement New feature or request review me PR is ready for review
Projects
Catalyst
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

🛠️ [TASK] : Enhance the rust builder to use techniques the same as the Earthly rust library
5 participants
@bkioshn @stevenj @minikin @Mr-Leshiy @apskhem