Problems building venvs from certain distributions. #1656

jsirois · 2022-03-07T22:36:56Z

@asherf reports two cases:

Exhibited by selenium 4.1.2

$ python3.8 -mpex selenium==4.1.2 --venv
Traceback (most recent call last):
  File "/home/jsirois/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jsirois/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/tmpyim2gyjr/__main__.py", line 86, in <module>
    bootstrap_pex(__entry_point__)
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/pex_bootstrapper.py", line 513, in bootstrap_pex
    venv_pex = ensure_venv(pex.PEX(entry_point, interpreter=target))
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/pex_bootstrapper.py", line 461, in ensure_venv
    shebang = populate_venv(
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/venv/pex.py", line 120, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/venv/pex.py", line 116, in record_provenance
    for src, dst in src_to_dst:
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/venv/pex.py", line 216, in _populate_deps
    for src, dst in record.reinstall(venv, symlink=symlink, rel_extra_path=rel_extra_path):
  File "/tmp/tmpyim2gyjr/.bootstrap/pex/pep_376.py", line 414, in reinstall
    raise ReinstallError(
pex.pep_376.ReinstallError: Cannot re-install file from /home/jsirois/.pex/installed_wheels/591b3b9ffc0450f97d2e0ef3559dd39b3259b707/selenium-4.1.2-py3-none-any.whl/selenium-4.1.2.dist-info/RECORD:1, refusing to install to absolute path /selenium/__init__.py.

Exhibited by greenlet 1.1.2

$ python3.8 -mpex greenlet==1.1.2 --venv
Traceback (most recent call last):
  File "/home/jsirois/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jsirois/.pyenv/versions/3.8.12/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/tmpb6vlpu58/__main__.py", line 86, in <module>
    bootstrap_pex(__entry_point__)
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/pex_bootstrapper.py", line 513, in bootstrap_pex
    venv_pex = ensure_venv(pex.PEX(entry_point, interpreter=target))
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/pex_bootstrapper.py", line 461, in ensure_venv
    shebang = populate_venv(
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/venv/pex.py", line 120, in populate_venv
    record_provenance(_populate_deps(venv, pex, venv_python, symlink))
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/venv/pex.py", line 116, in record_provenance
    for src, dst in src_to_dst:
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/venv/pex.py", line 216, in _populate_deps
    for src, dst in record.reinstall(venv, symlink=symlink, rel_extra_path=rel_extra_path):
  File "/tmp/tmpb6vlpu58/.bootstrap/pex/pep_376.py", line 423, in reinstall
    raise ReinstallError(
pex.pep_376.ReinstallError: Cannot re-install file from /home/jsirois/.pex/installed_wheels/673770872757fe48f96719d7818b2bffb6afba16/greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/greenlet-1.1.2.dist-info/RECORD:1, path does not match --target scheme: ../../../../home/jsirois/dev/pantsbuild/jsirois-pex/include/site/python3.8/greenlet/greenlet.h

The text was updated successfully, but these errors were encountered:

jsirois · 2022-03-08T00:00:51Z

The greenlet case seem to reveal a bug in pip install --target mode, but only when Pip detects its in a venv (and Pex runs Pip via a dogfood venv). In that case you get a leak of include that looks like so:

^jsirois@gill /tmp/workspace $ python3.8 -mvenv venv
^jsirois@gill /tmp/workspace $ source venv/bin/activate
(venv) ^jsirois@gill /tmp/workspace $ pip install --target /tmp/greenlet-chroot-pip-in-venv greenlet==1.1.2 --no-deps
Collecting greenlet==1.1.2
  Downloading greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB)
     |████████████████████████████████| 156 kB 2.0 MB/s 
Installing collected packages: greenlet
Successfully installed greenlet-1.1.2
WARNING: You are using pip version 21.1.1; however, version 22.0.4 is available.
You should consider upgrading via the '/tmp/workspace/venv/bin/python3.8 -m pip install --upgrade pip' command.
(venv) ^jsirois@gill /tmp/workspace $ ls
include  venv
(venv) ^jsirois@gill /tmp/workspace $ ls -l /tmp/greenlet-chroot-pip-in-venv/
total 0
drwxr-xr-x 5 jsirois jsirois 200 Mar  7 15:50 greenlet
drwxr-xr-x 2 jsirois jsirois 220 Mar  7 15:50 greenlet-1.1.2.dist-info
(venv) ^jsirois@gill /tmp/workspace $ head -2 /tmp/greenlet-chroot-pip-in-venv/greenlet-1.1.2.dist-info/RECORD 
../../../workspace/include/site/python3.8/greenlet/greenlet.h,sha256=63uaNbRd8ebE-dysD_SY2GwqbdRam2qSeSPfnaUNn6E,4245
greenlet-1.1.2.dist-info/AUTHORS,sha256=swW28t2knVRxRkaEQNZtO7MP9Sgnompb7B6cNgJM8Gk,849

Whereas running pip not in any venv gives the expected ../.. prefix for target mode and no leak:

(venv) ^jsirois@gill /tmp/workspace $ deactivate 
^jsirois@gill /tmp/workspace $ pip3.8 install --target /tmp/greenlet-chroot-pip-not-in-venv greenlet==1.1.2 --no-deps
Collecting greenlet==1.1.2
  Using cached greenlet-1.1.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (156 kB)
Installing collected packages: greenlet
Successfully installed greenlet-1.1.2
WARNING: You are using pip version 21.1.1; however, version 22.0.4 is available.
You should consider upgrading via the '/home/jsirois/.pyenv/versions/3.8.12/bin/python3.8 -m pip install --upgrade pip' command.
^jsirois@gill /tmp/workspace $ ls -l /tmp/greenlet-chroot-pip-not-in-venv/
total 0
drwxr-xr-x 5 jsirois jsirois 200 Mar  7 15:51 greenlet
drwxr-xr-x 2 jsirois jsirois 220 Mar  7 15:51 greenlet-1.1.2.dist-info
drwxr-xr-x 3 jsirois jsirois  60 Mar  7 15:51 include
^jsirois@gill /tmp/workspace $ head -2 /tmp/greenlet-chroot-pip-not-in-venv/greenlet-1.1.2.dist-info/RECORD 
../../include/python/greenlet/greenlet.h,sha256=63uaNbRd8ebE-dysD_SY2GwqbdRam2qSeSPfnaUNn6E,4245
greenlet-1.1.2.dist-info/AUTHORS,sha256=swW28t2knVRxRkaEQNZtO7MP9Sgnompb7B6cNgJM8Gk,849

Although there is no leak, the include path is also not correct for venv installation. It should be ../../include/site/python3.8/greenlet/greenlet.h; so the answer lies somewhere in the middle :/

The correct answer only presents when using the --prefix install scheme.

jsirois · 2022-03-08T00:02:06Z

The selenium case does not appear to be corrected by any combination of install scheme or venv / non-venv. The absolute path entries simply appear to be bogus and duplicate and ignored by Pip. Pex should do the same and maybe warn.

jsirois · 2022-03-08T00:45:57Z

Alright, I think the right way to proceed here is to switch from using pip install --target to using pip install --prefix. In order for the PEX zipapp scheme to work and in order to keep the interpreter a wheel happens to be installed with from affecting PEX reproducibility, a post install re-arrangement will be needed that satisfies the current zipapp scheme (site-packages content belongs in the chroot root). Upon reinstall into a venv, the re-arrangement will need to be reversed and pythonX.Y path components adjusted (this includes RECORD entries). FWICT this is both maximally correct and more robust. Pex will no longer have to know about install scheme semantics, it will just need to know how to reverse its own rearranging.

jsirois · 2022-03-09T05:26:13Z

Alright, I've got this working save for a handful of ITs and a handful of unit tests using --prefix. Bootstrapping vendoring with this new scheme was a bear, but that is sussed. I'll be putting in enough time tomorrow to get a PR and a Pex release out for this (jsirois@10b4400).

jsirois · 2022-03-10T01:46:42Z

Alright, the only remaining problem is pypy gets a different layout under pip prefix installs than cpython. There are two problems here:

The pypy layout is broken: the jupyter-server / nbconvert data files are simply not installed. This puts Pex back in the same boat as before the fix for that issue, but for pypy only.
Since the layout is different from that for cpython, reproducibility is broken when switching from building a PEX via pypy to cpython and vice-versa.

Still noodling on this, but it seems to me both are acceptable or will need to be.

jsirois · 2022-03-10T22:02:24Z

Alright, it turns out 1 was untrue, the pypy layout is not broken for the jupyter-server share/ data files case - it was just me wrongly assuming the nbconvert transitive dep carried those, when it was in fact ipykernel. I've managed to get all this working, but it's hefty in LOC. Simple idea, lots of layout details to get right.

Previously Pex used Pip's `--target` scheme which had both known bugs (pypa/pip#7658) and unknown quirks that Pex was failing to fully be able to work around. Switch to the `--prefix` scheme which exactly mirrors the scheme venvs use so that venvs can be created with content of all sorts placed where it belongs. This removes fragile parsing and interpretation of the RECORD; now Pex only creates a RECORD, which is much more straight forward, when building a venv. Fixes pex-tool#1656

Previously Pex used Pip's `--target` scheme which had both known bugs (pypa/pip#7658) and unknown quirks that Pex was failing to fully be able to work around. Switch to the `--prefix` scheme which exactly mirrors the scheme venvs use so that venvs can be created with content of all sorts placed where it belongs. This removes fragile parsing and interpretation of the RECORD; now Pex only creates a RECORD, which is much more straight forward, when building a venv. Partially addresses #1655 by switching to sha256 for all external artifact hashing. Only internal hashing remains for: 1. `interpreters` / INTERP-INFO 2. `venvs` and `unzipped_pexes` / PEX-INFO pex_hash (but this is a hash that includes all distributions' sha256 hashes). Fixes #1656 Closes #1662

jsirois added the bug label Mar 7, 2022

jsirois self-assigned this Mar 7, 2022

jsirois added the in progress label Mar 7, 2022

jsirois mentioned this issue Mar 10, 2022

Switch Pex installed wheels to --prefix scheme. #1661

Merged

This was referenced Mar 12, 2022

Release 2.1.72 #1663

Closed

Release 2.1.71 #1642

Closed

jsirois closed this as completed in #1661 Mar 12, 2022

jsirois removed the in progress label Mar 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems building venvs from certain distributions. #1656

Problems building venvs from certain distributions. #1656

jsirois commented Mar 7, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 9, 2022 •

edited

Loading

jsirois commented Mar 10, 2022

jsirois commented Mar 10, 2022

Problems building venvs from certain distributions. #1656

Problems building venvs from certain distributions. #1656

Comments

jsirois commented Mar 7, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 8, 2022

jsirois commented Mar 9, 2022 • edited Loading

jsirois commented Mar 10, 2022

jsirois commented Mar 10, 2022

jsirois commented Mar 9, 2022 •

edited

Loading