Incorrect script paths in RECORD for --target installations

[lkollar]

Environment

• pip version: 20.0.2
• Python version: CPython 3.8
• OS: Linux (RHEL 7.7)

Description

When using the --target option, pip rewrites the RECORD file and adds the relative paths to scripts in the bin directory if any were installed. However, the _handle_target_dir function in commands/install.py moves the bin directory into the target location, so it's adjacent to the installed packages in that folder. However, the RECORD file still contains the old relative path to the bin directory which is now incorrect.

Expected behavior

RECORD (and installed-files.txt) should contain the correct path to scripts installed in bin.

How to Reproduce

1. pip install --target /tmp/pip_test pip
2. head /tmp/asdf/pip_test/RECORD
3. Observe that the pip script is listed under ../../bin/pip instead of bin/pip

[chrahunt]

Did this ever work as expected?

@pradyunsg, any particular reason to mark this as 20.0?

[pradyunsg]

@pradyunsg, any particular reason to mark this as 20.0?

Nope. I think I did that accidentally, since I'd meant to add #7629.

---

pip 20.0.2:

$ pip install --target /tmp/pip_test pip
Collecting pip
  Using cached pip-20.0.2-py2.py3-none-any.whl (1.4 MB)
Installing collected packages: pip
Successfully installed pip-20.0.2
$ head /tmp/pip_test/**/RECORD
../../bin/pip,sha256=UpNtmEY-AzmiVJhbeHSbKeMrJOadJmd05v4ciEaXWbE,268
../../bin/pip3,sha256=UpNtmEY-AzmiVJhbeHSbKeMrJOadJmd05v4ciEaXWbE,268
../../bin/pip3.8,sha256=UpNtmEY-AzmiVJhbeHSbKeMrJOadJmd05v4ciEaXWbE,268
pip-20.0.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
pip-20.0.2.dist-info/LICENSE.txt,sha256=W6Ifuwlk-TatfRU2LR7W1JMcyMj5_y1NkRkOEJvnRDE,1090
pip-20.0.2.dist-info/METADATA,sha256=MSgjT2JTt8usp4Hopp5AGEmc-7sKR2Jd7HTMJqCoRhw,3352
pip-20.0.2.dist-info/RECORD,,
pip-20.0.2.dist-info/WHEEL,sha256=8zNYZbwQSXoB9IfXOjPfeNwvAsALAjffgk27FqvCWbo,110
pip-20.0.2.dist-info/entry_points.txt,sha256=HtfDOwpUlr9s73jqLQ6wF9V0_0qvUXJwCBz7Vwx0Ue0,125
pip-20.0.2.dist-info/top_level.txt,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4

pip 18.1:

$ pip install --target /tmp/pip_test pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-20.0.2
You are using pip version 18.1, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ head /tmp/pip_test/**/RECORD
../../bin/pip,sha256=DNLdfkdMEVHhyyvlrQqVKMBxJo04RfN0THMgDf0-v_w,271
../../bin/pip3,sha256=DNLdfkdMEVHhyyvlrQqVKMBxJo04RfN0THMgDf0-v_w,271
../../bin/pip3.8,sha256=DNLdfkdMEVHhyyvlrQqVKMBxJo04RfN0THMgDf0-v_w,271
pip-20.0.2.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
pip-20.0.2.dist-info/LICENSE.txt,sha256=W6Ifuwlk-TatfRU2LR7W1JMcyMj5_y1NkRkOEJvnRDE,1090
pip-20.0.2.dist-info/METADATA,sha256=MSgjT2JTt8usp4Hopp5AGEmc-7sKR2Jd7HTMJqCoRhw,3352
pip-20.0.2.dist-info/RECORD,,
pip-20.0.2.dist-info/WHEEL,sha256=8zNYZbwQSXoB9IfXOjPfeNwvAsALAjffgk27FqvCWbo,110
pip-20.0.2.dist-info/entry_points.txt,sha256=HtfDOwpUlr9s73jqLQ6wF9V0_0qvUXJwCBz7Vwx0Ue0,125
pip-20.0.2.dist-info/top_level.txt,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4

pip 18.0:

$ pip install --target /tmp/pip_test pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
Successfully installed pip-20.0.2
You are using pip version 18.0, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ head /tmp/pip_test/**/RECORD
pip/__init__.py,sha256=U1AM82iShMaw90K6Yq0Q2-AZ1EsOcqQLQRB-rxwFtII,455
pip/__main__.py,sha256=NM95x7KuQr-lwPoTjAC0d_QzLJsJjpmAoxZg0mP8s98,632
pip/_internal/__init__.py,sha256=j5fiII6yCeZjpW7_7wAVRMM4DwE-gyARGVU4yAADDeE,517
pip/_internal/build_env.py,sha256=--aNgzIdYrCOclHMwoAdpclCpfdFE_jooRuCy5gczwg,7532
pip/_internal/cache.py,sha256=16GrnDRLBQNlfKWIuIF6Sa-EFS78kez_w1WEjT3ykTI,11605
pip/_internal/configuration.py,sha256=MgKrLFBJBkF3t2VJM4tvlnEspfSuS4scp_LhHWh53nY,14222
pip/_internal/exceptions.py,sha256=6YRuwXAK6F1iyUWKIkCIpWWN2khkAn1sZOgrFA9S8Ro,10247
pip/_internal/legacy_resolve.py,sha256=L7R72I7CjVgJlPTggmA1j4b-H8NmxNu_dKVhrpGXGps,16277
pip/_internal/locations.py,sha256=VifFEqhc7FWFV8QGoEM3CpECRY8Doq7kTytytxsEgx0,6734
pip/_internal/main.py,sha256=IVBnUQ-FG7DK6617uEXRB5_QJqspAsBFmTmTesYkbdQ,437

[lkollar]

I'm happy to work on this but I'm not sure what the correct solution is. When RECORD gets written we don't know that we are doing a --target installation. The other option is to rewrite RECORD and installed-files.txt when _handle_target_dir is invoked but this doesn't seem very elegant as it needs to know implementation details about metadata formats where it really shouldn't.

Maybe the correct solution is to extract the logic of rewriting RECORD in a separate function and invoke it after the installation has been completed, as a second pass to fix up paths.

Any thoughts @pradyunsg?

[chrahunt]

There is a long-term refactoring underway to normalize the interface for installation and make all modes of installation take and respect a Scheme object which has all of the directories. If that is in place, then when the user specifies a --target directory we would map that to a Scheme object and, assuming all of the logic for installation respects the scheme that gets input, it should do everything correct with respect to installation and generating the RECORD file.

You can see #7309 (comment) for the next step of that refactoring. In order to do it we need to make some backwards-incompatible changes, so the options are to either get buy-in on the proposal in my comment or wait for 6 months so we can explicitly fail if provided with the now-deprecated --install-* options in --install-option.