Add scripts directory option when using pip install --target target_dir

[chhantyal]

• Pip version: 8.1.2
• Python version: 2.7.10_2
• Operating System: OSX 10.10.5

Description:

When I use --target option, I can specify target directory, however scripts are not inside. There is bit detailed StackOverflow question: http://stackoverflow.com/questions/26476607/how-do-you-specify-bin-directory-for-pip-install-with-target-option-enabled

It would be great if there was option to specify bin/ directory as well.

What I've run:

pip install ipython -t python_modules

ipython is installed inside python_modules but there is no bin/ipython

[techtonik]

Should be labelled with https://github.com/pypa/pip/issues?q=is%3Aissue+is%3Aopen+target+label%3A%22--target+option%22

[xavfernandez]

Slightly related to #3980

[AlexeySanko]

I agree that it would be great. But how to implement it? It's sub-option of --target option and cannot be used without it.
To add option --target-scripts? Users try to use it without --target option.
To move into new command install_target/target from install? We will have problems with backward compatibility.

[pickfire]

Is there any workaround on this where we could install a binary wheel to a specific target with the binaries installed?

[munkyboy]

a workaround is to use the --user option. i.e. pip install --user flake8. This will put scripts in ~/.local/bin

[gokcennurlu]

I'd like to work on this (PyPA Sprint Weekend at Bloomberg).

[brainwane]

@gokcennurlu Great! How's this going?

[DavidBord]

@pradyunsg following our conversation on this issue at pycon, what I understand from the code is that scripts are not copied to target on purpose, and it is reference to #5201. Please advice whether you want to add this functionality or not.

[evanunderscore]

The original problem here seems to have been fixed by the change in #4152; the bin directory does end up being copied to the target directory now. Unsure if this was intentional since the related change seemed to only be trying to address data_files. Perhaps #5203 (in response to the aforementioned #5201) should be reverted now?

[tstaig]

From our perspective having the bin directory inside the directory pointed by --target is better, but still not enough.

We do make use of virtual envs, however on top of that we use a patching system through PATH/PYTHONPATH variable (and other variables, bear in mind that in our context Java, Python and C++ coexist and communicate through CORBA). The --prefix <path> flag would be enough, however our legacy code installs in different <path1>/lib/python/site-packages, <path2>/lib/python/site-packages, etc., inestead of <path1>/lib/python3.8/site-packages, <path2>/lib/python3.8/site-packages, etc., which has the positive effect of being able to dynamically swap between Python versions.

The use of --target works for libraries, which are placed correctly at --target <path>/lib/python/site-packages, however having the bin directory at <path>/lib/python/site-packages/bin doesn't make sense and we have to manually copy/symlink the executable scripts to <path>/bin directory.

Either allowing to provide the scripts location for the --target flag (something like --target-scripts) or providing a mean to change the site-packages location (lib/pythonX.Y/, lib/python/, Lib/, etc.) for the --prefix would work for us, but at the moment we do see either alternative to the old --install-option as incomplete.

In previous versions of Pip we made use of the '--install-option="--prefix=<path>" --install-option="--install-purelib=<path>/lib/python/site-packages"' flags, however newer versions don't allow to use these. And even in the old ones, that meant to disable the use of wheels and compile everything locally, which was definitely not ideal.

Is this issue still being worked on or should we go ahead and try to workaround the situation locally by cp/ln of the scripts in the bin directory or even by patching Pip itself?

[tstaig]

In case anyone is interested, I added the --target-scripts flag with some small changes to the pip/_internal/commands/install.py file. The diff is from an older version of Pip though (20.1.1), but it should be easily adapted to the latest version.

--- pip.orig/_internal/commands/install.py	2021-09-08 20:49:01.552217116 +0000
+++ pip/_internal/commands/install.py	2021-09-08 21:22:38.731729919 +0000
@@ -108,6 +108,13 @@
                  '<dir>. Use --upgrade to replace existing packages in <dir> '
                  'with new versions.'
         )
+        cmd_opts.add_option(
+            '--target-scripts',
+            dest='target_scripts_dir',
+            metavar='dir',
+            default=None,
+            help='Install packages\' scripts into <dir>. It requires the -t or --target flag to be used.'
+        )
         cmdoptions.add_target_python_options(cmd_opts)
 
         cmd_opts.add_option(
@@ -240,6 +247,9 @@
         if options.use_user_site and options.target_dir is not None:
             raise CommandError("Can not combine '--user' and '--target'")
 
+        if options.target_scripts_dir is not None and options.target_dir is None:
+            raise CommandError("Can not use '--target-scripts' without '--target'")
+
         cmdoptions.check_install_build_global(options)
         upgrade_strategy = "to-satisfy-only"
         if options.upgrade:
@@ -273,6 +283,15 @@
             target_temp_dir = TempDirectory(kind="target")
             target_temp_dir_path = target_temp_dir.path
 
+            if options.target_scripts_dir is not None:
+                options.target_scripts_dir = os.path.abspath(options.target_scripts_dir)
+                if (os.path.exists(options.target_scripts_dir) and not
+                        os.path.isdir(options.target_scripts_dir)):
+                    raise CommandError(
+                        "Target's scripts path exists but is not a directory, will not "
+                        "continue."
+                    )
+
         global_options = options.global_options or []
 
         session = self.get_default_session(options)
@@ -446,12 +465,12 @@
 
         if options.target_dir:
             self._handle_target_dir(
-                options.target_dir, target_temp_dir, options.upgrade
+                options.target_dir, target_temp_dir, options.target_scripts_dir, options.upgrade
             )
 
         return SUCCESS
 
-    def _handle_target_dir(self, target_dir, target_temp_dir, upgrade):
+    def _handle_target_dir(self, target_dir, target_temp_dir, target_scripts_dir, upgrade):
         ensure_dir(target_dir)
 
         # Checking both purelib and platlib directories for installed
@@ -464,12 +483,19 @@
             scheme = distutils_scheme('', home=target_temp_dir.path)
             purelib_dir = scheme['purelib']
             platlib_dir = scheme['platlib']
+            scripts_dir = scheme['scripts']
             data_dir = scheme['data']
 
+            if target_scripts_dir is None:
+                target_scripts_dir = os.path.join(target_dir, os.path.relpath(scripts_dir, target_temp_dir.path))
+            ensure_dir(target_scripts_dir)
+
             if os.path.exists(purelib_dir):
                 lib_dir_list.append(purelib_dir)
             if os.path.exists(platlib_dir) and platlib_dir != purelib_dir:
                 lib_dir_list.append(platlib_dir)
+            if os.path.exists(scripts_dir):
+                lib_dir_list.append(scripts_dir)
             if os.path.exists(data_dir):
                 lib_dir_list.append(data_dir)
 
@@ -479,7 +505,10 @@
                         ddir = os.path.join(data_dir, item)
                         if any(s.startswith(ddir) for s in lib_dir_list[:-1]):
                             continue
-                    target_item_dir = os.path.join(target_dir, item)
+                    if lib_dir == scripts_dir:
+                        target_item_dir = os.path.join(target_scripts_dir, item)
+                    else:
+                        target_item_dir = os.path.join(target_dir, item)
                     if os.path.exists(target_item_dir):
                         if not upgrade:
                             logger.warning(

[tstaig]

If anyone is interested, I also prepared a pull request for latest version that you can find in
#10472

[matham]

A problem I ran into and why I'm interested in this option, is that we provide a python dmg containing a venv for macos built as universal binaries. We provide this to users so they can download extract the dmg, install their additional packages with pip, build a new dmg from it and distribute it as a GUI app.

Because we provide it built using universal binaries, users can install their additional packages with pip natively on intel or apple silicon. One use case is for someone to run it on intel (when they don't have a M1 available, e.g. on the CI) but install with pip install ... --platform macosx_11_0_arm64 --only-binary=:all: --target venv/lib/python*/site-packages. This will install arm64 wheels only to the extracted dmg. Then, they can build a new dmg and it'll run on arm64.

When using --platform pip forces that --target must also be set. This means that e.g. we set --target venv/lib/python*/site-packages, but now we have a bin folder in site-packages, when ideally it would be under venv/bin.

[Spindel]

I've got the same use-case as @matham here, wanting to target armv7 with a specific python-version from automated builds. Until recently it's all gone okay, but a change in setuptools + deprecation means we need to downgrade setuptools to install one package, and doing that on build-machine to target something else felt bad, so we went for a virtualenv.

Now that turned into a magnificent disaster of a debugging ride, and let's just say, I'd really want this feature to work. Any way to override the script path, as long as I can do it, would be very very good and welcome.

[ChrisHills463]

Just to add another use case, I would like to install a package for different python versions (cpython 3.9 and pypy 3.9). If I install one after the other, the original scripts are overwritten. Probably not relevant to this specific bug, but another way to solve it for my case would be to add a suffix to the installed scripts (e.g. --target-scripts-suffix=_pypy3.9). This could ideally be a global config setting somewhere so binaries automatically get the extension if none is supplied to pip.