Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluate PEX_ROOT cache structure. #1655

Closed
jsirois opened this issue Mar 7, 2022 · 3 comments · Fixed by #2513
Closed

Evaluate PEX_ROOT cache structure. #1655

jsirois opened this issue Mar 7, 2022 · 3 comments · Fixed by #2513
Assignees
Labels
tech-debt Issue that addresses technical debt.

Comments

@jsirois
Copy link
Member

jsirois commented Mar 7, 2022

There are a few things to evaluate:

  1. Versioning: Likely the cache should be versioned to allow clean incompatible changes to structure.
  2. Hashing: cache keys have been built off sha1 content hashes since day1. Analyze attack vulnerability and see if sha256 could be used in (most) places. There are some places, maybe only in venvs, where path length becomes a problem due to shebang length limits.
  3. Scavenging: What protions of the cache can be scavenged without affecting performance of X too badly. This is effectively about coming up with a pex3 cache tool that allows inspection and selective pruning.
@jsirois jsirois added the tech-debt Issue that addresses technical debt. label Mar 7, 2022
@jsirois
Copy link
Member Author

jsirois commented Mar 11, 2022

It turns out #1661 had to deal with 1 (versioning the installed_wheels portion of the cache and all the other portions that symlinked into it (venvs and unzipped_pexes)) and 2 (switching to sha256 hashes for installed_wheels) solved most of 1 in its particular case. Generally we'll still need cache versioning though. You can only change hash algorithms so often!

@jsirois
Copy link
Member Author

jsirois commented Mar 11, 2022

I broke out #1662 for item 2 since that looks closeable.

@jsirois jsirois removed their assignment Mar 11, 2022
jsirois added a commit that referenced this issue Mar 12, 2022
Previously Pex used Pip's `--target` scheme which had both known bugs
(pypa/pip#7658) and unknown quirks that Pex
was failing to fully be able to work around. Switch to the `--prefix`
scheme which exactly mirrors the scheme venvs use so that venvs can be
created with content of all sorts placed where it belongs.

This removes fragile parsing and interpretation of the RECORD; now Pex
only creates a RECORD, which is much more straight forward, when
building a venv.

Partially addresses #1655 by switching to sha256 for all external
artifact hashing. Only internal hashing remains for:
1. `interpreters` / INTERP-INFO
2. `venvs` and `unzipped_pexes` / PEX-INFO pex_hash (but this is a hash
   that includes all distributions' sha256 hashes).

Fixes #1656
Closes #1662
jsirois added a commit that referenced this issue Aug 13, 2024
Change the default `PEX_ROOT` cache location from `~/.pex` to
`<OS default user cache dir>/pex`.

Work towards #2201, #1655 and #1176.
@jsirois jsirois self-assigned this Aug 14, 2024
@jsirois jsirois closed this as completed in 84a4196 Sep 4, 2024
@jsirois
Copy link
Member Author

jsirois commented Sep 4, 2024

Item, 1 is now directly addressed with versioned cache entries like this:

pex/pex/cache/dirs.py

Lines 93 to 95 in 84a4196

INSTALLED_WHEELS = Value(
"installed_wheels",
version=0,

Item 3 is mainly addressed with the pex3 cache {dir,info,purge} commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
tech-debt Issue that addresses technical debt.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant