pip install --user tries to uninstall from system site-packages

[cdeil]

• Pip version: 9.0.1
• Python version: 3.5
• Operating system: Mac (Macports)

Description:

I've been a happy user of pip for a while, thank you for all your work!

Now (possibly after a recent pip update), the bahaviour has changed and

pip install . --user

from a source folder in a Python project I work on (Gammapy) tries to uninstall a stable 0.5 version of Gammapy I have in the Macports site-packages.

I wasn't expecting pip to do this uninstall and it's not clear to me from the pip 9.0 and 9.0.1 release notes that this is an intended change in behaviour.

Is this a bug or am I doing it wrong?
What's the workaround?

What I've run:

$ pip --version
pip 9.0.1 from /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (python 3.5)
$ pip install . --user
Processing /Users/deil/code/gammapy
Requirement already satisfied: numpy>=1.8 in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: astropy>=1.3 in /Users/deil/Library/Python/3.5/lib/python/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: regions in /Users/deil/Library/Python/3.5/lib/python/site-packages (from gammapy==0.6.dev4330)
Requirement already satisfied: click in /opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages (from gammapy==0.6.dev4330)
Installing collected packages: gammapy
  Found existing installation: gammapy 0.5
    Uninstalling gammapy-0.5:
Exception:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 544, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/gammapy' -> '/var/folders/sb/4qv5j4m90pz1rw7m70rj1b1r0000gn/T/pip-lc_5uj1j-uninstall/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/gammapy'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/commands/install.py", line 342, in run
    prefix=options.prefix_path,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 778, in install
    requirement.uninstall(auto_confirm=True)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_install.py", line 754, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 556, in move
    rmtree(src)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: '__init__.py'

[piotr-dobrogost]

https://pip.pypa.io/en/stable/reference/pip_install/?highlight=ignore#cmdoption-i

[cdeil]

@piotr-dobrogost - Yes, I managed to do what I wanted using this:

pip install . --user --no-deps -I

So there is certainly a workaround.

The question from my side remains -- is this intended / good behaviour by pip?

[pfmoore]

It's intended, because Python (not pip, but Python's core import mechanisms) doesn't really like multiple versions of a package on sys.path. Consider if foo version 1.0 included foo/bar.py but foo version 2.0 removed it. You could then still import foo, foo.bar, but would get the v2.0 version of foo and the v1.0 version of bar.

Basically, pip uninstalls older versions precisely to avoid this issue. The --ignore flag says "I know what I'm doing, and take responsibility for any issues caused by having both versions on sys.path together".

[cdeil]

This is a recent change in behaviour from pip though, no?
(to try to uninstall stuff in system site-packages when I say pip install something --user)

Was it very recently?
Is it visible in the pip release notes?
Am I the first one to be puzzled by this?

Personally I'm fine with the old or new behaviour.

So just close this issue?
Or is there something that can / should be improved about the pip release notes or docs?
Is there a way to "follow" such updates / changes in pip, like an RSS feed or blog?

[pfmoore]

TBH, I'm not sure about that. I never put packages in both places, so I can't honestly say from my own experience, and I'd have to trawl the changelogs to see if there was any specific mention.

I'm not sure if you're the first one to be caught by this - I know I've explained the "you shouldn't have multiple versions on sys.path" issue before, but I can't recall if it was in relation to this or some other scenario.

If you propose a change to pip, leave the issue open and your suggestion can be looked at - but my personal view is that the current behaviour is fine (which I think you're OK with). If you think a doc change would help, we'd be very grateful for a PR with your suggested rewording. If we missed including something in the changelogs, then it's probably water under the bridge now (I don't know if we have a policy on retroactively updating the changelog). Going forward, the changelog is definitely where you'd keep up to date with what's changed as new versions come out.

[cdeil]

I don't know if it's a good idea to start uninstalling stuff when a user says install. My guess is that there will be surprises and gotchas either way.

But if that's the intended behaviour, fine. I'm still surprised I haven't encountered this before and didn't have to use --ingore-installed for the past few years where I've been using pip.

I had a look at the docs and found it hard to find a description of the behaviour of pip.
Here it says

Install the packages (and uninstall anything being upgraded/replaced).
https://pip.pypa.io/en/stable/reference/pip_install/?highlight=ignore-installed#overview

Searching for ignore-installed I found this:
https://pip.pypa.io/en/stable/user_guide/?highlight=ignore-installed#user-installs

pip install --user follows four rules:

1. When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.
2. When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv).

It's not obvious to me from that description that pip will basically always try to uninstall the given package I'm trying to install, if it's anywhere on my sys.path. Maybe that section could be the place to point that behaviour out more explicitly?
(there already is an example using --ignore-installed at the very bottom of the section)

Or maybe there could be another mention of this behaviour and of --ignore-installed already earlier in the pip docs? Mentioning that pip install will try to uninstall stuff and will commonly fail if one doesn't have write premissions for system packages could even go in the quickstart IMO.

@pfmoore and @piotr-dobrogost - Close issue now? Or do you agree that some docs edits (which ones exactly) could be made to improve the description?

[pfmoore]

@piotr-dobrogost seems to have found some of the history (although for some reason I can only see his comment in email, not here). Looks like it is changed behaviour, and was requested by some people. Agreed it should be documented better (but I don't understand the original details of the change, so can't really say how to describe the current situation). Sorry, I can't really offer more help.

[piotr-dobrogost]

I agree behavior of install --user is not intuitive to say the least. Well, to say the truth I think it's plainly wrong. I remember making comments to this effect in the past here. As the whole user scheme itself is defined so that its location comes before the location of site-packages on sys.path there should be no check performed of what's already installed in site-packages when using user scheme for installation. The only thing worth considering is what to do when the package had already been installed using user scheme. Although it's not clear for me if it's even technically possible to establish what's installed in user scheme vs what's installed in site-packages; otherwise the problem such as How to uninstall a package installed with pip install --user wouldn't have existed I guess. However I guess that after recent change in setuptools and finally getting rid of sys.path mangling performed in easy_install.pth it should be safe just to install in user scheme no matter if it's already installed there.

[piotr-dobrogost]

@pfmoore
You should immediately delete this email as I deleted my comment which you received in this email :) I deleted the comment as it was talking about editable install not user scheme one.

[pfmoore]

Ah, that's what happened! Sorry. Github emails don't reflect edits, I've been bitten by that before...

[pfmoore]

Agreed what happens is unintuitive, and I wouldn't be opposed if someone argued it's wrong. However, as I noted, having 2 different versions in system site-packages and user site-packages could cause issues. We can't win either way - someone's going to be bitten by unintuitive behaviour no matter what we do.

Ideally, someone should review this whole area and propose a resolution - which may be documentation only or may include behaviour changes. But without any funded support for pip, that's not likely to happen unless a user who's actually experiencing issues steps up to do the work.

[piotr-dobrogost]

(...) having 2 different versions in system site-packages and user site-packages could cause issues.
It should not as the whole point of user site-packages is to take precedence over global site-packages; the two are meant to be distinct and separate namespaces. The only problem is pip's wrong implementation of install --user.

[pfmoore]

It should not as the whole point of user site-packages is to take precedence over global site-packages

Well, yes - but that's not how Python works. We can happily reject bug reports as "not a pip issue", but that still leaves us with confused users.

If you want to raise a bug against Python, I guess you could. I doubt it'd be easy to get it fixed, though.

[cdeil]

The behaviour I had for the past years and got used to was that pip install never uninstalls, but just puts new versions. And then somehow Python import always picks up the newest. It was OK for me. The new, current behaviour is also reasonable, other package managers do the same thing to remove older versions before putting the new one.

I probably won't participate further in discussions and don't have time to contribute code or docs fixes to pip, sorry.

So just to summarise, from the user perspective, my main wish here is to get more information about how pip works and when new releases come out, to explain better what behaviour changed (via a "What's new page" or blog post).
One presentation I found very interesting was http://www.dabeaz.com/modulepackage/ which explains about Python modules and packages and imports. If someone here has time to put together a similar "deep dive" detailed talk or tutorial for pip in the future, I would find that super interesting.

[piotr-dobrogost]

Well, yes - but that's not how Python works.

I see, you are referring here to the contrived example with foo.bar you gave earlier. Well, I didn't think of this corner case earlier. This indeed justifies considering globally installed package when installing the package in user scheme. Albeit I would say it's a bug in Python to take foo from one place and foo.bar from another not related to the first. Upon importing foo the import logic should lock import location for all subsequent imports beginning with foo. Now, when I'm aware of this unfortunate behavior of import mechanics in Python I'm wondering what were the authors of user scheme thinking when they created the scheme?

[pfmoore]

They probably assumed people wouldn't install in both system packages and user packages. Or maybe they just didn't think of this issue. Who knows? And yes, my example was contrived. However, I think (although I can't be sure, or provide references) that we've had real issues as a result of this.

On a purely personal note, I only use the system site-packages (I'm on Windows, and I have a per-user install of Python 3.6, so this doesn't affect anything else like a system package manager) and virtualenvs, so the issue doesn't affect me either way (apart from needing to provide support to people on issues like this). So I'll leave this discussion now to be taken further by people who are affected.

[piotr-dobrogost]

@dstufft
How is user scheme meant to work in light of problem described in #4337 (comment) ?

[pfmoore]

@piotr-dobrogost It works fine if you install a given package in only user or system location, not in both. Sorry I may have misunderstood as I assumed that you realised that. It's only when you try to install different versions the same distribution in both places that there's any possibility of an issue.

[pfmoore]

@piotr-dobrogost My apologies. Over on distutils-sig, @ncoghlan corrected me on this. Apparently the import system does protect against multiple version clashes like this. At least to the extent that the first version that's on sys.path will always "win", so there's never going to be inconsistent imports like I described.

I couldn't find anything in the docs about whether the user scheme comes before or after the system scheme in sys.path. I assume it comes before, so that user installs take precedence over system installs, but I've been bitten once today by not testing my claims, so I'll just say "I assume" here... :-(

[dstufft]

Yea, it goes:

1. stdlib always wins
2. local directory (i.e. .).
3. user directory
4. site-packages

In each of those, I think any .pth files get added immediately after the directory that introduced the .pth file.

[piotr-dobrogost]

Over on distutils-sig, @ncoghlan corrected me on this.

Can you share link to this?

Apparently the import system does protect against multiple version clashes like this.

Ok. In that case we are back to pip's behavior and my earlier statement that pip shouldn't care what's already installed globally when installing in user scheme. @dstufft Would you agree?

[pfmoore]

https://mail.python.org/pipermail/distutils-sig/2017-March/030281.html

[mrmachine]

#4466 is possibly a dupe. Reading the pip docs (https://pip.pypa.io/en/stable/user_guide/#user-installs), it seems very clear to me as a user that:

1. When globally installed packages are on the python path, and they conflict with the installation requirements, they are ignored, and not uninstalled.

means pip install --user ... never uninstalls. Globally installed packages (in system site-packages) should be ignored, not uninstalled, and the new package should be installed in the user scheme which will be found and imported first.

So this appears to be a regression in 9.x and in contradiction with the docs.

As far as I understand it, the whole point of the user scheme is that users may not have control over system packages, and may not be able to uninstall anything from there.

Even if they did have permission, one user installing a new version of a package in their user namespace, triggering an uninstall from system site-packages, would break that package for all other users on that system.

[piotr-dobrogost]

Globally installed packages (in system site-packages) should be ignored (...)
They should but unless you pass --ignore-installed that's sadly not the case which is very confusing. In practice pip does not see difference between packages installed in user namespace and in system site-packages.

[mrmachine]

@piotr-dobrogost but if I use --ignore-installed, all my requirements will be (re)installed into the userbase directory. Not only the ones that aren't already satisfied by what is already installed in system site-packages. Again, this is in contradiction with the docs, and is different to previous behaviour of pip?

2. When globally installed packages are on the python path, and they satisfy the installation requirements, pip does nothing, and reports that requirement is satisfied (similar to how global packages can satisfy requirements when installing packages in a --system-site-packages virtualenv).

When the first two of four rules for the way a feature operates are completely wrong, and the remaining rules are basically just sanity checks preventing action where it wouldn't make sense anyway, it seems there's not much point in supporting the user scheme at all.

I'm a bit confused though, about why I'm pretty sure this was working as expected and as documented before, but is now completely broken.

[pradyunsg]

Related #4575

[reynoldsnlp]

@cdeil would it be an improvement to have pip send a message on stderr that tells the user that it is deleting the older version? That seems like a simple fix that would make it clear to the user what is happening. That message would also give users that don't want this behavior a string to search on Google that would help them find the -I flag.

[pradyunsg]

The --ignore-installed flag isn't a exact fix for the problem here.

#4575 and #1668 are the proper fixes for this and will need someone to either do it or fund work to get it done. (https://wiki.python.org/psf/Fundable%20Packaging%20Improvements)

[cdeil]

Ha ha, I haven't used --user since 2 years, switched to use conda envs and conda and pip install things there and that works fine. So personally I don't really care about --user any more.

But re-reading the discussion above: yeah, I would still say a PermissionError from system site packages when trying to pip install --user something is a bug. Similar to what others said above, pip install --user should just install the package and not look at all what's in system site-packages. I think the initial counter-argument that foo.bar gets mixes is not true, Python doesn't do that, it's OK to have the same package installed multiple times in different envs in the sys.path list, like in this case system site-packages and now user site-packages.

@cdeil would it be an improvement to have pip send a message on stderr that tells the user that it is deleting the older version?

Well, yes, if that is an easy addition, and the real fix and change in behaviour of pip install --user to not delete isn't coming within the near future (say ~ 1 year), then you could add that message now.