Scripts installed with wrong shebang under subprocess

[jaraco]

On Python 3.5.1 but not on 2.7.10, when I invoke pip through a subprocess in a virtualenv, the scripts are created with the wrong shebang and thus fail to run:

$ cat > invoke.py
import sys
import subprocess

subprocess.Popen(sys.argv[1:]).wait()
$ rm -Rf env
$ python -m virtualenv --version
14.0.0
$ python --version
Python 3.5.1
$ python -m virtualenv env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
Also creating executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python
Installing setuptools, pip, wheel...done.
$ python invoke.py env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Library/Frameworks/Python.framework/Versions/3.5/bin/python3

One would expect the py.test script to have a shebang with 'env' in it.

Using pyvenv instead is not subject to the issue. Removing the __PYVENV_LAUNCHER__ environment variable before launching the subprocess works around the issue, as reported here. The issue was observed with 13.1.2 and 14.0.0.

Is this behavior expected? If so, is removing the environment variable the appropriate thing for a parent process to do to work around the issue?

[Ivoz]

does

$ env/bin/pip uninstall pytest
$ env/bin/pip install pytest
$ head -n 1 env/bin/py.test

Give same, or different results?

And same question with

$ env/bin/python -m pip install pytest

?

[Ivoz]

Possibly a regression somehow on #322 / #541

In which case, could you test whether

subprocess.Popen(sys.argv[1:], env={}).wait()

helps or not?

[Ivoz]

Hmm, there was code in pip / distlib to perhaps deal with this, but it got commented out a year ago-

https://github.com/pypa/pip/blob/7cea748d0fb2e8980c2676304607426585550b41/pip/_vendor/distlib/util.py#L157-L165

[jaraco]

Give same, or different results?

In both cases, invoking pip directly results in a proper shebang:

$ env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.29 in ./env/lib/python3.5/site-packages (from pytest)
Installing collected packages: pytest
Successfully installed pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
$ env/bin/pip uninstall -y pytest
Uninstalling pytest-2.8.5:
  Successfully uninstalled pytest-2.8.5
$ env/bin/python -m pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): py>=1.4.29 in ./env/lib/python3.5/site-packages (from pytest)
Installing collected packages: pytest
Successfully installed pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python

Also, invoking with an empty env works:

$ cat invoke.py 
import sys
import os
import subprocess

env = dict(os.environ)
env.pop('__PYVENV_LAUNCHER__')
env = {}

subprocess.Popen(sys.argv[1:], env=env).wait()
$ python -m virtualenv env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python3
Also creating executable in /Users/jaraco/Dropbox/code/public/aspen/env/bin/python
Installing setuptools, pip, wheel...done.
$ python invoke.py env/bin/pip install pytest
Collecting pytest
  Using cached pytest-2.8.5-py2.py3-none-any.whl
Collecting py>=1.4.29 (from pytest)
  Using cached py-1.4.31-py2.py3-none-any.whl
Installing collected packages: py, pytest
Successfully installed py-1.4.31 pytest-2.8.5
$ head -n 1 env/bin/py.test
#!/Users/jaraco/Dropbox/code/public/aspen/env/bin/python3

[jaraco]

According to the commit, that workaround was commented with inclusion of distlib 0.2.0 released in pip 6.0.

[jaraco]

That commit references pip 2031.

[jaraco]

Also, I've installed Python from the python.org Mac installer, not through homebrew or other methods.

[jaraco]

The code was removed from distlib in this commit, which provides little rationale for the change.

[Ivoz]

I guess it's to be "expected" given that your initial $ python is setting __VENV_LAUNCHER__ and thereafter fooling other scripts / executables. Unfortunately I don't have OS X to debug how exactly that process of "fooling" goes on.

Also could be similar / same issue as was reported in #620

@jaraco it could be worth modifying a pip in a virtualenv to uncomment the mentioned distlib lines and seeing if that seems to fix pip's behaviour of picking up the wrong python to write

[jaraco]

I traced that venv detection code back to this commit, which unfortunately doesn't elucidate the origins of the idea.

[jaraco]

Seems nobody knows the meaning of this env var.

[jaraco]

I've found this explanation.

[jaraco]

My inclination was similar to what Ronald suggested here.

[Ivoz]

@vsajip could you give any advice?

[vsajip]

Some thoughts:

• __PYVENV_LAUNCHER__ was added because for earlier versions of Python (< 3.3, according to Ronald) but not for later versions, Python's site.py on OS X needed to distinguish the location of stub launchers from framework executables. According to this comment by Ned Deily, sys.executable now points to the stub launcher, and this is why distlib stopped using __PYVENV_LAUNCHER__. The only usage of it now is in site.py - if available, on OS X, it's used to locate the pyvenv.cfg file.
• It may be worth checking to see what happens in e.g. Python 3.4, to see if's some sort of regression, or whether all Python >= 3.3 has the same behaviour.
• My guess is that something, somewhere, is doing an os.path.realpath() call when it shouldn't be. It doesn't look like it's distlib doing that. If (as per Ned's comment) sys.executable and the env var are always the same, then it shouldn't matter; but if something does a realpath on a sys.executable and executes Python through the result, then that would result in an unexpected shebang (because sys.executable would then be a dereferenced path).

[tdsmith]

A(nother) simple demonstration of the problem, which Homebrew rediscovered in Homebrew/homebrew-core#8129:

$ virtualenv -p python3 test
$ test/bin/python3 -c 'import sys; print(sys.executable)'
/Users/tim/test/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess; subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; print(sys.executable)"])'
/usr/local/bin/python3

$ /usr/local/bin/python3 -c 'import subprocess, os; del os.environ["__PYVENV_LAUNCHER__"]; subprocess.call(["/Users/tim/test/bin/python3", "-c", "import sys; print(sys.executable)"])'
/Users/tim/test/bin/python3

[boxed]

I was just bitten by this exact bug. It's quite a severe bug in my opinion, mostly because it's so extremely confusing and hard to figure out when it hits.

[jaraco]

I've been hit by what I think is a more severe manifestation of this issue.

I've started using xonsh as my daily shell.

But when I try to use virtualenv under xonsh, it's unusable:

~ $ cd ~/draft
draft $ virtualenv --version
16.0.0
draft $ virtualenv .env
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/jaraco/draft/.env/bin/python3
Also creating executable in /Users/jaraco/draft/.env/bin/python
Installing setuptools, pip, wheel...done.
draft $ .env/bin/pip install paver
Collecting paver
  Using cached https://files.pythonhosted.org/packages/98/1e/37ba8a75bd77ea56a75ef5ae66fe657b79205bbc36556c9456cd641782a4/Paver-1.3.4-py2.py3-none-any.whl
Collecting six (from paver)
  Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six, paver
Successfully installed paver-1.3.4 six-1.11.0
  The script paver is installed in '/Users/jaraco/draft/.env/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
draft $ .env/bin/paver --help
Traceback (most recent call last):
  File ".env/bin/paver", line 7, in <module>
    from paver.tasks import main
ModuleNotFoundError: No module named 'paver'
draft $ head -n 1 .env/bin/paver
#!/Library/Frameworks/Python.framework/Versions/3.7/bin/python3

All installed scripts seem to be getting the system prefix and not the virtualenv prefix.

This probably stems from the fact that xonsh is running under Python 3.7 (system prefix) and so has this in the env:

draft $ env | grep LAUNCHER
__PYVENV_LAUNCHER__=/Library/Frameworks/Python.framework/Versions/3.7/bin/python3

If I instead launch zsh and run the same commands or if I first invoke del $__PYVENV_LAUNCHER__, paver runs as expected and the shebang line points to the virtualenv.

Should xonsh be clearing that environment variable when it starts up?

[nsoranzo]

There is a possible bugfix at python/cpython#9516

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Just add a comment if you want to keep it open. Thank you for your contributions.

[retracile]

I still see this same behavior on Mojave with Python 3.7.6 (installed via Homebrew).

[gaborbernat]

Can you check with the rewrite branch?

[retracile]

@gaborbernat Attempting to build the rewrite branch (fbdd782) gives me a build error from python3 setup.py build.

...src/virtualenv/__init__.py", line 3, in <module>
    from .version import __version__
ModuleNotFoundError: No module named 'virtualenv.version'

which looks unrelated to this issue.

[pfmoore]

Attempting to build the rewrite branch (fbdd782) gives me a build error from python3 setup.py build.

You should use pip to do the build - pip wheel . - as the rewrite branch uses pyproject.toml to define the build process.

[gaborbernat]

The error you're getting is setuptools way of saying I don't support the new standard way of building python libraries (PEP-517/518). It's a setuptools bug at best, but as @pfmoore pointed you should use pip to build a wheel, or even better to point it to the virtualenv folder to install it (it will automatically build a wheel and install it in one go).

[retracile]

Testcase:

#!/bin/bash
rm -rf venv-test
virtualenv --python python3 venv-test
python3 -c 'import subprocess; subprocess.check_call(["./venv-test/bin/pip3", "install", "markdown"])'
shebang=$(head -1 venv-test/bin/markdown_py)
expected_shebang="#!$(pwd)/venv-test/bin/python"
if [ "$shebang" == "$expected_shebang" ]; then
    echo "PASSED"
else
    echo "FAILED: \"$shebang\" != \"$expected_shebang\""
    exit 1
fi

This fails under python3+virtualenv 16.7.9, and passes under python3+virtualenv-16.7.10.dev11+gfbdd782 (ie, the rewrite branch).

[gaborbernat]

Fixed in the rewrite as per above.