pipenv install --deploy --system doesn't respect PIP_IGNORE_INSTALLED=1 in 2020.8.13

[judgeaxl]

Issue description

When requesting pip to install in user mode, and also to ignore any installed packages, (PIP_USER=1 PIP_IGNORE_INSTALLED=1) pipenv will still skip system installed packages if they meet the requirements. This worked fine in 2020.6.2.

While this use case might seem a bit obscure at first, it's a great way to install and generate local wheels in a multi-step Dockerfile ensuring that everything is included in a single folder that can easily be copied over to a minimal python image.

Expected result

I expect all packages to be installed in the expected location, regardless of if they exist at the system level, when PIP_IGNORE_INSTALLED is set.

Actual result

Any system package meeting the requirements are skipped and not installed.

Steps to replicate

I'm using Docker here to avoid contaminating the local user installation, but the same happens outside of Docker.

In a new folder, run pipenv install six to create a Pipfile with six as the only dependency.

Using this Dockerfile in the same folder, run docker build . -t test

FROM python:3.7-buster
WORKDIR /build
# System install, which includes six
RUN pip install pipenv==2020.8.13

COPY Pipfile* ./
RUN env PIP_USER=1 PIP_IGNORE_INSTALLED=1 pipenv install --deploy --system

Use docker run --rm -ti test /bin/bash and verify that six.py has not been installed into /root/.local in the container.

To verify that this is not something pip does, add the following line at the end of the Dockerfile, repeat the steps above, and notice that it correctly installs six in /root/.local

RUN env PIP_USER=1 PIP_IGNORE_INSTALLED=1 pip install six

(This is the equivalent of running pip install --user --ignore-installed six)

Workaround

Using the previous version 2020.6.2 of pipenv solves the problem form now.

RUN pip install pipenv==2020.6.2

[judgeaxl]

I believe this defect is related: #4276

[haizaar]

Happens to me as well. Here is the self-contained reproduction:

$ cat Dockerfile
FROM python:3.8.5-alpine3.12
ARG PIPENV_VER

RUN pip install pipenv==$PIPENV_VER

RUN mkdir /build
WORKDIR /build

ENV PYROOT /pyroot
ENV PYTHONUSERBASE $PYROOT
RUN mkdir /pyroot

RUN pipenv install google-api-python-client
RUN rm -rf /root/.local/share/virtualenv*

RUN PIP_USER=1 PIP_IGNORE_INSTALLED=1 pipenv install --system --deploy
RUN ls /pyroot/lib/python3.8/site-packages/six*  # That itself is weird

$ docker build . --build-arg PIPENV_VER=2020.6.2 -t pipenv-bug
<finishes fine>

$ docker build . --build-arg PIPENV_VER=2020.8.13 -t pipenv-bug
...
ls: /pyroot/lib/python3.8/site-packages/six*: No such file or directory
The command '/bin/sh -c ls /pyroot/lib/python3.8/site-packages/six*' returned a non-zero code: 1

I.e. the six package wasn't installed since it was already present on the system and PIP_IGNORE_INSTALLED was ignored.

[frostming]

Using PIP_ env vars to control the behavior of Pipenv is not documented and tested. And it is desired to skip satisfied dependencies(#4216).

And if you are curious, it used to work on 2020.6.2 because of a bug #4315 that is fixed on 2020.8.13.

So it is not encouraged to depend on hidden features. You should submit a feature request if current (documented) behavior doesn't meet your requirement.

[judgeaxl]

If so, it's probably good to update the final part of this section of the documentation to be more clear what pip environment variables are supported and which aren't.

https://pipenv.pypa.io/en/latest/advanced/#configuration-with-environment-variables

Excerpt:

Also note that pip itself supports environment variables, if you need additional customization.

For example:

$ PIP_INSTALL_OPTION="-- -DCMAKE_BUILD_TYPE=Release" pipenv install -e .

[judgeaxl]

While I think it would be a useful feature of pipenv, to either support this natively, or to respect the env variable, I can think of an alternate approach:

pipenv lock -r --ignore-outdated >requirements.txt
pip install --user --ignore-installed -r requirements.txt

As far as I can tell this will create the virtualenv, but not install anything, and then generate a requirements.txt file which I can give to pip install with all my custom flags.

[judgeaxl]

@frostming I misread your comment earlier; that using PIP_ variables to control pipenv is not supported makes sense, despite the documentation I referred to. But it's still perhaps something that would be good to highlight in the docs.

[haizaar]

@frostming I understand where you are coming from, but this approach was working since Pipenv inception and seems to be used quite a lot and not just by me: #3160.

I understand the value of doing things right. But there is also a great value of deprecation and migration paths. Breaking something that was working for years without any notice and clear migration path is painful as both this and #4432 clearly show.

And currently there is still no clear migration path to achieve the same:

1. The above workaround with generating requirements, from my understanding, will not support private pypi repos defined in the Pipefile (personally, I don't use that feature currently but we try to generic here)
2. Using virtualenv will increase image size by 13MB (mainly because of pip and setuptools), which is 30% of the total image in my case. And I'd rather not to have package manager available on the the production images.

What's your opinion on the above?

[haizaar]

Btw, with pipenv lock -r approach, setuptools somehow still ends up as part of the installed packages (+3.5mb)

[judgeaxl]

Using PIP_ env vars to control the behavior of Pipenv is not documented and tested. And it is desired to skip satisfied dependencies(#4216).

@frostming OK, sorry to keep beating at this, but for some reason I didn't see this in the documentation before, which directly contradicts your statement:

To ensure that all pip-installable components actually are installed into the virtual environment and system packages are only used for interfaces that don’t participate in Python-level dependency resolution at all, use the PIP_IGNORE_INSTALLED setting:

$ PIP_IGNORE_INSTALLED=1 pipenv install --dev

[judgeaxl]

Btw, with pipenv lock -r approach, setuptools somehow still ends up as part of the installed packages (+3.5mb)

@haizaar You can redirect the virtual env to another location, using WORKON_HOME or delete .local/share/virtualenv* before you copy. This might still leave some extra stuff around, but is at least somewhat better.

[haizaar]

Thank you Andreas for digging that documentation piece. I had a feeling I'm not completely crazy here. Regarding WORKON_HOME override - how does it differ from "venv-in-project" approach? Using venv (and copying but over to the final image) takes even more space since it includes pip as well which another 10mb. Plus, I don't want to have package installer in my production image.

…

On Mon, 14 Sep 2020, 00:31 Andreas Axelsson, ***@***.***> wrote: Btw, with pipenv lock -r approach, setuptools somehow still ends up as part of the installed packages (+3.5mb) @haizaar <https://github.com/haizaar> You can redirect the virtual env to another location, using WORKON_HOME or delete .local/share/virtualenv* before you copy. This might still leave some extra stuff around, but is at least somewhat better. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4453 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOGWNHMU5RHDWQRFFNPQLSFTJVHANCNFSM4QUJLAZA> .

[judgeaxl]

@haizaar It looks like when using WORKON_HOME pip and setuptools are still put in ~/.local/share/virtualenv, so yes, one would still have to delete or exclude those manually.

But I tried one more workaround, where I instead use PYTHONUSERBASE=/new/empty/dir to get the installed packages out of ~/.local, and I believe that gives a pretty clean folder to copy over. I've not counted bytes, this is just giving it a cursory glance.

[Gyllsdorff]

Using PIP_ env vars to control the behavior of Pipenv is not documented and tested. And it is desired to skip satisfied dependencies(#4216).

@frostming OK, sorry to keep beating at this, but for some reason I didn't see this in the documentation before, which directly contradicts your statement:

Here's a link to the Pipenv documentation where it uses PIP_IGNORE_INSTALLED in a example, that suggests that PIP_IGNORE_INSTALLED is a supported env variable.
https://github.com/pypa/pipenv/blob/c945f916b68f3fc46dd9a971a9fe574e797a5e7b/docs/advanced.rst#-working-with-platform-provided-python-components

[frostming]

I am hesitating to do this in Pipenv while it shouldn't be difficult to support.

It's true pip's env var may work when Pipenv doesn't take in the charge because pip is used internally by Pipenv to install packages. But when Pipenv implements its own logic, things get different: if we are to support PIP_IGNORE_INSTALLED, what about other env vars that are not documented nor tested? And people may rely on this and regard it as a regression issue. Examples are PIP_USER, PIP_INSTALL_TARGET and so on. Or we are going to only recognize this one and ignore others.

[uranusjr]

I believe we already recommend users to use pip options in some cases, so it’d be reasonable for pipenv to explicitly offer support a number of pip options. But IMO PIP_IGNORE_INSTALLED specifically should not be one of them—even pip maintainers want it to go away, and we shouldn’t make lives harder.

TBH I don’t understand why people feel so strongly about using virtual environments in a Docker container. Environments created by virtualenv 20 have no runtime penalty at all (and the build time delay is minimal if you disable auto update), and the end result is much more simple and deterministic. But maybe we can add a new flag to install that syncs packages to a prefix of user choice (like pip’s --prefix), which is easy to manage for pip maintainers, and cover all usages above.

[judgeaxl]

@uranusjr, there's a couple of reasons for not wanting to use a virtualenv, that I can think of, there might be others.

The ultimate goal is to get the Docker image size down as much as possible, and since you're highly likely to need compilers and other tooling in the image where you run pipenv install, you almost certainly want to copy the resulting installed artifacts to a slim image afterwards.

Having pip, setuptools and other base modules that come with a virtualenv in your production container is another thing people want to avoid for various reasons, e.g. size, and just to minimize potential misuse. My own image is large enough that I can see past that, but to @haizaar it apparently made a significant difference.

A quick test for Python 3.7 shows there's around 8.5MiB of tooling in a fresh venv, using python:3.7-buster as the base image, none of which my app needs to run. Sure the base image with the OS layers is a lot bigger, but we're now trending more and more to distro-less and blank containers for running apps, so in the end everything counts.

Installing to the user folder makes sense because in a fresh Docker image you wouldn't have anything else conflicting, so it's known to be clean, and thus suitable for a 1:1 copy.

So besides the size/tooling issue, using a virtualenv is more complicated as you need to somehow pass on the location to the runtime image so you know where to copy from. I guess that can either be done by using PIPENV_VENV_IN_PROJECT to get a fixed location (./.venv), or through simply moving the resulting virtualenv to some fixed location before copying it. I'm not aware that you can read anything from the base image and use it as arguments to the Dockerfile COPY command (which is used to copy files from one image to another), so just relying on the pipenv --venv output there wouldn't work.

I'm currently reasonably happy with my requirements.txt solution which does all I need and which results in a minimal installation, so to me the remaining Pipenv problem is that the documentation suggests usage patterns which don't actually work anymore, i.e. using PIP_IGNORE_INSTALLED via Pipenv.

Should pip eventually decide to deprecate PIP_IGNORE_INSTALLED, then I guess I have no other "simple" option but to go with the virtualenv solution, and live with the slight overhead.

[matteius]

Maybe I am missing something, but I don't see PIP_IGNORE_INSTALLED in the pip code base going back all the versions I checked. There is a flag ignore_installed that is present for many versions now, but I don't see where it is impacted by an environment variable. I only find reference to PIP_IGNORE_INSTALLED is two rst files of pipenv. What am I missing here?

[judgeaxl]

I imagine pip programmatically matches PIP_ + UPPERCASE_OPTION_NAME_WITH_UNDERSCORE_FOR_DASHES for the environment, rather than spelling them all out in source, since I believe pretty much all options can be specified using the environment.

[judgeaxl]

https://pip.pypa.io/en/stable/topics/configuration/#environment-variables