Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce # of resolves when packaging pex_binary and python_awslambda targets #12343

Merged
merged 1 commit into from
Jul 14, 2021

Conversation

Eric-Arellano
Copy link
Contributor

@Eric-Arellano Eric-Arellano commented Jul 14, 2021

We already use repository PEXes when [python-setup].resolve_all_constraints or experimental lockfile are set, thanks to code in pex_from_targets.py. Those benefit from using the same repository PEX as the rest of the codebase, e.g. when using MyPy or Pytest.

However, we were ignoring this repository pex with python_awslambda and pex_binary and resolving more granular repository PEXes after having already resolved the global one. Iiuc, those more granular ones are not coming from the global one. This is wasted work.

While a TwoStepPex could be useful in those two callsites if you are not using constraints/lockfiles, we strongly encourage using lockfiles.

Removing TwoStepPex allows us to remove complexity in a hairy part of the codebase, especially as we add new complexity for lockfile support.

[ci skip-rust]
[ci skip-build-wheels]

# Rust tests and lints will be skipped. Delete if not intended.
[ci skip-rust]

# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
@Eric-Arellano Eric-Arellano requested review from stuhood and benjyw July 14, 2021 01:49
@Eric-Arellano
Copy link
Contributor Author

Eric-Arellano commented Jul 14, 2021

I think we may actually want to cherry-pick this to 2.6.x? It looks like this is resulting in more work than necessary, like we were setting up multiple repository PEXes (which is how the code reads to me). Compare before:

❯ ./pants package build-support/bin::
There is no pantsd metadata at /Users/eric/code/pants/.pids/48089be3df2a/pantsd.
18:50:08.87 [INFO] Initializing scheduler...
18:50:09.82 [INFO] Scheduler initialized.
18:50:12.07 [INFO] Completed: Building build-support.bin/deploy_to_s3.pex
18:50:12.12 [INFO] Completed: Building build-support.bin/check_inits.pex
18:50:12.14 [INFO] Completed: Building build-support.bin/check_banned_imports.pex
18:50:12.70 [INFO] Completed: Resolving 2 requirements: ansicolors==1.1.8, typing-extensions==3.7.4.3
18:50:12.81 [INFO] Completed: Resolving 1 requirement: packaging==20.9
18:50:13.24 [INFO] Completed: Resolving 4 requirements: ansicolors==1.1.8, requests[security]>=2.25.1, types-requests==2.25.0, typing-extensions==3.7.4.3
18:50:13.29 [INFO] Completed: Building build-support.bin/reversion.pex with 2 requirements: ansicolors==1.1.8, typing-extensions==3.7.4.3
18:50:13.39 [INFO] Completed: Building build-support.bin/changelog.pex with 1 requirement: packaging==20.9
18:50:13.82 [INFO] Completed: Building build-support.bin/release_helper.pex with 4 requirements: ansicolors==1.1.8, requests[security]>=2.25.1, types-requests==2.25.0, typing-extensions==3.7.4.3
18:50:15.85 [INFO] Completed: Resolving 4 requirements: PyYAML<5.5,>=5.4, toml==0.10.2, types-PyYAML==5.4.3, types-toml==0.1.3
18:50:16.23 [INFO] Completed: Building build-support.bin/generate_github_workflows.pex with 4 requirements: PyYAML<5.5,>=5.4, toml==0.10.2, types-PyYAML==5.4.3, types-toml==0.1.3
18:50:18.12 [INFO] Completed: Resolving 12 requirements: PyYAML<5.5,>=5.4, ansicolors==1.1.8, packaging==20.9, psutil==5.8.0, pystache==0.5.4, requests[security]>=2.25.1, setproctitle==1.2.2, toml==0.10.2, types-PyYAML==5.4.3, typ... (66 characters truncated)
18:50:19.28 [INFO] Completed: Building build-support.bin/generate_docs.pex with 12 requirements: PyYAML<5.5,>=5.4, ansicolors==1.1.8, packaging==20.9, psutil==5.8.0, pystache==0.5.4, requests[security]>=2.25.1, setproctitle==1.2.2... (106 characters truncated)
18:50:19.29 [INFO] Wrote dist/build-support.bin/changelog.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/check_banned_imports.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/check_inits.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/deploy_to_s3.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/generate_docs.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/generate_github_workflows.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/release_helper.pex
18:50:19.29 [INFO] Wrote dist/build-support.bin/reversion.pex

After:

❯ ./pants package build-support/bin::
There is no pantsd metadata at /Users/eric/code/pants/.pids/48089be3df2a/pantsd.
18:49:06.73 [INFO] Initializing scheduler...
18:49:07.74 [INFO] Scheduler initialized.
18:49:21.49 [INFO] Completed: Resolving 3rdparty/python/lockfile.txt
18:49:22.42 [INFO] Completed: Building build-support.bin/deploy_to_s3.pex
18:49:22.42 [INFO] Completed: Building build-support.bin/check_banned_imports.pex
18:49:22.45 [INFO] Completed: Building build-support.bin/changelog.pex with 1 requirement: packaging==20.9
18:49:22.45 [INFO] Completed: Building build-support.bin/check_inits.pex
18:49:22.46 [INFO] Completed: Building build-support.bin/generate_github_workflows.pex with 4 requirements: PyYAML<5.5,>=5.4, toml==0.10.2, types-PyYAML==5.4.3, types-toml==0.1.3
18:49:22.46 [INFO] Completed: Building build-support.bin/reversion.pex with 2 requirements: ansicolors==1.1.8, typing-extensions==3.7.4.3
18:49:22.81 [INFO] Completed: Building build-support.bin/release_helper.pex with 4 requirements: ansicolors==1.1.8, requests[security]>=2.25.1, types-requests==2.25.0, typing-extensions==3.7.4.3
18:49:23.53 [INFO] Completed: Building build-support.bin/generate_docs.pex with 12 requirements: PyYAML<5.5,>=5.4, ansicolors==1.1.8, packaging==20.9, psutil==5.8.0, pystache==0.5.4, requests[security]>=2.25.1, setproctitle==1.2.2... (106 characters truncated)
18:49:23.54 [INFO] Wrote dist/build-support.bin/changelog.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/check_banned_imports.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/check_inits.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/deploy_to_s3.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/generate_docs.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/generate_github_workflows.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/release_helper.pex
18:49:23.54 [INFO] Wrote dist/build-support.bin/reversion.pex

@Eric-Arellano Eric-Arellano changed the title [internal] Remove TwoStepPex abstraction Reduce # of resolves when packaging pex_binary and python_awslambda targets Jul 14, 2021
@Eric-Arellano Eric-Arellano merged commit d4cf50e into pantsbuild:main Jul 14, 2021
@Eric-Arellano Eric-Arellano deleted the two-step branch July 14, 2021 03:21
Eric-Arellano added a commit to Eric-Arellano/pants that referenced this pull request Jul 14, 2021
We already use repository PEXes when `[python-setup].resolve_all_constraints` or experimental lockfile are set, thanks to code in `pex_from_targets.py`. Those benefit from using the same repository PEX as the rest of the codebase, e.g. when using MyPy or Pytest.

However, we were ignoring this repository pex with `python_awslambda` and `pex_binary` and resolving more granular repository PEXes after having already resolved the global one. Iiuc, those more granular ones are not coming from the global one. This is wasted work. 

While a `TwoStepPex` could be useful in those two callsites if you are not using constraints/lockfiles, we strongly encourage using lockfiles. 

Removing `TwoStepPex` allows us to remove complexity in a hairy part of the codebase, especially as we add new complexity for lockfile support.

[ci skip-rust]
[ci skip-build-wheels]
Eric-Arellano added a commit that referenced this pull request Jul 15, 2021
@wisechengyi wisechengyi mentioned this pull request Jul 17, 2021
wisechengyi added a commit that referenced this pull request Jul 17, 2021
### Internal

* [internal] Manually fix Black lockfile to handle interpreter constraints ([#12366](#12366))
* Revert "Prefix the entire setup.py chroot. (#12359)" ([#12370](#12370))
* [internal] Cache native client binary in CI ([#12355](#12355))
* Prefix the entire setup.py chroot. ([#12359](#12359))
* [internal] Fix AWS CLI breaking due to Python 2 usage ([#12364](#12364))
* [Internal] Add `git_url()` helper to `docutil.py` ([#12352](#12352))
* [Internal] Refactor how `PythonToolBase` exposes requirements and interpreter constraints ([#12356](#12356))
* [internal] Add experimental per-tool lockfiles with Docformatter as an example ([#12346](#12346))
* Remove Pants's dpeendency on `requests` ([#12348](#12348))
* [internal] Remove `TwoStepPex` abstraction ([#12343](#12343))
* Add psycopg2-binary to default module mapping ([#12339](#12339))
* [internal] Upgrade toolchain pants plugin to 0.13.1 ([#12338](#12338))
* Add an API to coarsen/partition Targets by their cycles ([#12251](#12251))
* Prepare 2.5.1 ([#12329](#12329))
* Bootstrap fewer JVM versions in Coursier/javac tests to hopefully reduce CI flakiness ([#12325](#12325))
* Native client respects `--concurrent`. ([#12324](#12324))
* Add client lib tests. ([#12322](#12322))
* Special case enum option parse failures. ([#12281](#12281))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants