pip --prefix tries to uninstall system packages

[casparvl]

(N.B. I wasn't sure if to submit this as a bug report, or a feature request, but I guess since the current behaviour is not contradicting the docs, it's not strictly speaking a bug)

What's the problem this feature will solve?
I'm administrating a shared Unix cluster. On this cluster, we provide a python installation with a number of packages on the system. We also have groups of users (think: research groups, students & teachers in the same course, etc) that may want to install additional Python packages in a shared prefix, so that the entire group has access to the same python packages. Finally, on top of that, user might want to install some private packages (i.e. essentially making 3 hierarchical levels of python installations: system, shared prefix, private prefix).

To create shared installations, we recommend that users use the pip install --prefix option, to install in a common prefix to which they all have access, and set the PYTHONPATH and PATH accordingly.

The issue is that pip --prefix tries to uninstall packages that are present on the system, if a newly installed packages requires a newer version. Example:

$ pip install --prefix my_shared_folder jupylet
...
Installing collected packages: tqdm, glfw, dataclasses, stringcase, marshmallow, mypy-extensions, typing-inspect, marshmallow-enum, dataclasses-json, gltflib, PyGLM, qtpy, qtconsole, jupyter-console, jupyter, glcontext, moderngl, soundfile, ipyevents, sounddevice, pillow, mido, parso, jedi, multipledispatch, pyrr, pyglet, moderngl-window, webcolors, jupylet
  WARNING: The script tqdm is installed in 'JHL_installations/Python//bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script jupyter-console is installed in 'JHL_installations/Python//bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Attempting uninstall: parso
    Found existing installation: parso 0.6.2
    Uninstalling parso-0.6.2:
ERROR: Could not install packages due to an EnvironmentError: [Errno 30] Read-only file system: 'AUTHORS.txt'

In this example, Parso-0.6.2 is installed in our system installation, but Parso>=0.7.0 is required for the given jupylet. But, because the user cannot touch the system installations, he/she cannot install on top of this system stack using --prefix.

Describe the solution you'd like
I would like pip install --prefix to stop trying to uninstall packages if the user running the pip --prefix does not have write access to those installations. To me, it seems using --prefix for user-space installations is quite common, so not having write access to the system installation is a pretty common case.

In other words: the desired behaviour is similar to what pip install --user already does. Note that it's also similar to what a pip install in a virtual environment (created with --system-site-packages) does, since that will skip uninstalling system packages, and warn the user about this.

My suggestion would be to make the behaviour similar to the one for the virtual environment: skip uinstalling, but do notify the user with a warning (especially since it's a change in the default behaviour).

If a change in default behaviour is completely undesirable, an additional, explicit flag (e.g. --no-uninstall) would also be an option. The downside is that this means that even packages that are writeable would not be uninstalled, which may also not be the desired effect.

Alternative Solutions
Alternative 1: pip install --prefix my_shared_folder jupylet --ignore-installed completes the installation succesfully, but has the disadvantage that it overwrites all of the system packages - even the ones which do satisfy jupylet's dependency constraints. This is overkill, and undesirable for other reasons (our system site packages are very well optimized for our hardware architecture, so we prefer them to be used whenever possible).

Alternative 2: I've also tried to combine --prefix with --user, hoping that this would allow me to do a 'user space installation' as described in https://pip.pypa.io/en/stable/user_guide/#user-installs but in a custom prefix. That, however, is not allowed, since --user also set's a prefix.

Alternative 3: PYTHONUSERBASE=my_shared_folder pip install jupylet --user works. However, it feels 'dirty' and seems to deviate quite a bit from what https://www.python.org/dev/peps/pep-0370/ describes (which is really a private-user installation). Additionally, I wouldn't want to set this PYTHONUSERBASE by default for these users, as they may want to also install stuff only for their own, local user (and a --user installing in a custom prefix is very unexpected for users).

Finally, if alternative 3 would - in your view - be the best solution for my case, it honestly makes me wonder what --prefix is for... Setting --prefix feels like a much more natural way (much closer also to e.g. Automake's --prefix, which also aims to satisfy user space installations).

Additional context
I think this may be related to what was discussed here #4575 though I'm not sure I completely grasp the proposals in that issue - and if they would accomodate my use case.

[uranusjr]

This is basically by design. --prefix is an overlay on the current environment, not replacing it—that’s what virtual environments are for. It’s analogous to --user and --target. Of course, it still should not uninstall the package in another environment (but simply install the needed version into the prefix), but that’s for #4575 to sort out.

[uranusjr]

Merging this into #4575.