Non eager updates ignored if the target flag is specified

[davidedelvento]

• Pip version: 9.0.1
• Python version: 2.7.7
• Operating system: all

Description:

I'm trying to install globus-sdk which has these already satisfied depdendencies:

$ pip list --format=columns | grep six
six             1.10.0  
$ pip list --format=columns | grep requests
requests        2.3.0   

Installing it as:

$ pip install --upgrade-strategy=only-if-needed globus-sdk                                                                                                                                  

(blah blah)

Requirement not upgraded as not directly required: requests<3.0.0,>=2.0.0 in /glade/u/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages (from globus-sdk)
Requirement not upgraded as not directly required: six==1.10.0 in /glade/u/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages (from globus-sdk)

(blah blah)

So in this case is working fine. However I need to install globus-sdk in another directory....

What I've run:

$ pip install --upgrade-strategy=only-if-needed --target=/glade/scratch/ddvento/test_installs/globus-sdk/ globus-sdk                                                                     
Collecting globus-sdk
  Using cached globus-sdk-0.6.0.tar.gz
Collecting requests<3.0.0,>=2.0.0 (from globus-sdk)
/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/glade/apps/opt/python/2.7.7/gnu-westmere/4.8.2/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached requests-2.13.0-py2.py3-none-any.whl
Collecting six==1.10.0 (from globus-sdk)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: requests, six, globus-sdk
  Running setup.py install for globus-sdk ... done
Successfully installed globus-sdk-0.6.0 requests-2.13.0 six-1.10.0

Et voila' the upgrade strategy is ignored and the already-satisfied requirements are reinstalled, causing a lot of troubles (e.g. when I upgrade them from their proper location, I have to chase down and upgrade them in all the other copied locations)

ddvento@yslogin5 /glade/apps/contrib/modulefiles $ ls -l /glade/scratch/ddvento/test_installs/globus-sdk/
total 768
drwxrwxr-x 8 ddvento consult    512 Mar 30 09:49 0.6.0
drwxrwxr-x 5 ddvento consult 131072 Mar 30 10:00 globus_sdk
drwxrwxr-x 2 ddvento consult    512 Mar 30 10:00 globus_sdk-0.6.0-py2.7.egg-info
drwxrwxr-x 3 ddvento consult 131072 Mar 30 10:00 requests
drwxrwxr-x 2 ddvento consult    512 Mar 30 10:00 requests-2.13.0.dist-info
drwxrwxr-x 2 ddvento consult    512 Mar 30 10:00 six-1.10.0.dist-info
-rw-rw-r-- 1 ddvento consult  30098 Mar 30 10:00 six.py
-rw-rw-r-- 1 ddvento consult  29545 Mar 30 10:00 six.pyc
drwxrwxr-x 5 ddvento consult    512 Mar 30 10:00 tests
                        

[pfmoore]

The intended use for --target is to install a set of requirements to a location that's intended to be complete without considering the system locations (for scenarios like vendoring, or building a zipapp). So by design --target doesn't consider the system site packages location. Your use case is somewhat different, as you want to add the --target`` directory to sys.path``` alongside the existing entries. That's probably not a use case that we've considered (because it's not entirely cleat why anyone would need to do this). So supporting this would probably be a new feature, and a clearer description of why you're doing what you are would help in understanding the need.

[davidedelvento]

Thanks for taking the time to explaining the use case of --target to me.

What option (or strategy) am I supposed to use to install packages in package-specific location, while still using the rest of the installed packages, without making each package installation a duplication of everything?

To be more clear, we have a machine shared among thousands of users, and each one needs a possibly different combinations of packages (particularly of package's versions, since different versions have different bugs), so I install each package in its own package/version directory and use PYTHONPATH to allow users to pick-and-choose.

[pfmoore]

I don't really know, TBH. It's not a use case that has come up before. I'm not sure I'd agree with the approach of having each package in its own directory and handling it via explicit PYTHONPATH manipulation, but I don't know your environment. In the situation you describe you could use --no-deps, I guess, as you don't seem to want the dependencies installed in the same place as the dependents.

It may simply be that pip doesn't have a good means to do what you want, at the moment. You seem to be taking a pretty unique approach to managing Python installations.

[dstufft]

I'm going to close this since it is a duplicate of #4116.