pip3 need honor sys.executable in OS X

[xu-cheng]

I found that pip3 from homebrew handles the shebang of installed scripts in wrong way(Homebrew/legacy-homebrew#32254). After further digging, I'm pretty sure it's pip's issue rather than homebrew.

The related code is in here:

def get_executable():
    if sys.platform == 'darwin' and ('__PYVENV_LAUNCHER__'
                                     in os.environ):
        result =  os.environ['__PYVENV_LAUNCHER__']
    else:
        result = sys.executable
    return result

I don't know why you guys would overwrite sys.executable. But as the test showed, under OS X python3 __PYVENV_LAUNCHER__ will always exist.

$ python
Python 2.7.8 (default, Aug 24 2014, 21:26:19)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;os.environ['__PYVENV_LAUNCHER__']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.8_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: '__PYVENV_LAUNCHER__'

$ python3
Python 3.4.1 (default, Aug 24 2014, 21:32:40)
[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;os.environ['__PYVENV_LAUNCHER__']
'/usr/local/bin/python3'

[tdsmith]

I think this is wheel-specific.

The desired behavior in Homebrew is that the shebang is set to the opt path, /usr/local/opt/python3/bin/python3.4, which does not change between micro versions of python (and not the Cellar path, /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4, which does). sys.executable is set to the opt path.

Installing ipython with pip3 --no-use-wheel gives #!/usr/local/opt/python3/bin/python3.4 as the shebang; installing ipython from the wheel uses #!/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4 as the shebang.

We would prefer that the wheel behavior matches the setup.py behavior.

[dstufft]

This is actually in the distlib project, or at least the linked code is. That's located at https://bitbucket.org/pypa/distlib.

Perhaps @vsajip can jump in and give a rationale though.

[tdsmith]

I would have been more inclined to blame fix_script in pip/wheel.py -- https://github.com/pypa/pip/blob/1.5.6/pip/wheel.py#L67-L88 -- except that I think it does what I want.

Is there some reason python3 would be invoked without applying sitecustomize during a wheel install?

[dstufft]

Not unless pip itself is invoked without it.

[tdsmith]

I think @xu-cheng was right; os.environ['PYENV_LAUNCHER'] takes the value of sys.argv[0] in python3; changing the shebang in the pip3 script changes the shebangs it writes for wheels.

I wonder why this is wheel-specific!

[vsajip]

The reason for the logic in distlib is that on OS X, a stub launcher is used which essentially loads the real interpreter in a framework build. The __PYVENV_LAUNCHER__ variable points to the stub, which is located next to pyvenv.cfg in a venv, and this allows the venv sys.path setup machinery to operate correctly.

The environment variable is set by the stub launcher, and so won't be present if the interpreter in the framework build is invoked directly.

@xu-cheng AFAIK, the executable pointed to by __PYVENV_LAUNCHER__ will, when run, invoke the appropriate framework build interpreter, so in what way exactly is this causing a problem?

[xu-cheng]

@vsajip As for my test shown, __PYVENV_LAUNCHER__ will always exist for python3 on OS X.

When using Homebrew we want the shebang set as sys.executable. But currently it's somehow set by __PYVENV_LAUNCHER__. The desired behavior in Homebrew is that the shebang is set to the opt path, /usr/local/opt/python3/bin/python3.4, which does not change between micro versions of python. And that's what sys.executable does. But now as set by __PYVENV_LAUNCHER__, the shabang is pointed to the Cellar path, /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4. So every time python gets micro update or a openssl security update occurs, the Cellar path will change which results the scripts cannot find the interpreter. That's the problem.

[vsajip]

@xu-cheng Unfortunately I am unable to build Python 3.4 on the somewhat aging Mac (Leopard) that I have access to. However, I do have a build which I made in January, and installed using sudo make altinstall. I ran some code three times:

1. Using the version installed on the machine using sudo make altinstall.
2. From a venv (venv33) created using virtualenv, using the Python in variant 1.
3. From a venv (venv33a) created using pyvenv, using the Python in variant 1.

I printed sys.executable and the os.environ keys:

vinay@iMac:projects$ python3.3
Python 3.3.3+ (3.3:72560f9bb2a2, Jan 14 2014, 12:36:39) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.executable, sorted(os.environ)
('/usr/local/bin/python3.3', ['Apple_PubSub_Socket_Render', 'COMMAND_MODE', 'DISPLAY', 'GPG_AGENT_INFO', 'HOME', 'LANG', 'LOGNAME', 'MANPATH', 'OLDPWD', 'PATH', 'PS1', 'PWD', 'SECURITYSESSIONID', 'SHELL', 'SHLVL', 'SSH_AUTH_SOCK', 'TERM', 'TERM_PROGRAM', 'TERM_PROGRAM_VERSION', 'TMPDIR', 'USER', 'WORKON_HOME', '_', '__CF_USER_TEXT_ENCODING'])
>>> ^D
vinay@iMac:projects$ venv33/bin/python
Python 3.3.0+ (3.3:459a23083b66, Dec 31 2012, 16:26:43) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.executable, sorted(os.environ)
('/Users/administrator/projects/venv33/bin/python', ['Apple_PubSub_Socket_Render', 'COMMAND_MODE', 'DISPLAY', 'GPG_AGENT_INFO', 'HOME', 'LANG', 'LOGNAME', 'MANPATH', 'OLDPWD', 'PATH', 'PS1', 'PWD', 'SECURITYSESSIONID', 'SHELL', 'SHLVL', 'SSH_AUTH_SOCK', 'TERM', 'TERM_PROGRAM', 'TERM_PROGRAM_VERSION', 'TMPDIR', 'USER', 'WORKON_HOME', '_', '__CF_USER_TEXT_ENCODING'])
>>> ^D
vinay@iMac:projects$ venv33a/bin/python
Python 3.3.0+ (3.3:459a23083b66, Dec 31 2012, 16:26:43) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.executable, sorted(os.environ)
('/Users/administrator/projects/venv33a/bin/python', ['Apple_PubSub_Socket_Render', 'COMMAND_MODE', 'DISPLAY', 'GPG_AGENT_INFO', 'HOME', 'LANG', 'LOGNAME', 'MANPATH', 'OLDPWD', 'PATH', 'PS1', 'PWD', 'SECURITYSESSIONID', 'SHELL', 'SHLVL', 'SSH_AUTH_SOCK', 'TERM', 'TERM_PROGRAM', 'TERM_PROGRAM_VERSION', 'TMPDIR', 'USER', 'WORKON_HOME', '_', '__CF_USER_TEXT_ENCODING', '__PYVENV_LAUNCHER__'])
>>> os.environ['__PYVENV_LAUNCHER__']
'/Users/administrator/projects/venv33a/bin/python'
>>> 

I know this is an old machine and an old Python version, but that shouldn't matter.

From the above, you'll see that __PYVENV_LAUNCHER__ is only in the environment when run from a venv created using pyvenv. Note also that in that case, the value of sys.executable and os.environ[__PYVENV_LAUNCHER__] are one and the same.

I'm also unfortunately unable to install a Python 3.4 on this machine from a python.org installer, so it's hard for me to look into it further. From what I can see, __PYVENV_LAUNCHER__ is not always in os.environ, at least in my environment (which has MacPorts BTW, though I am not using it as a source for Python interpreters).

[xu-cheng]

I wonder why the second and third results of your test have different python version with your first one.

Anyway, I installed python 3.3.0 on my Mac through pyenv (not confused with pyvenv). Here's the results:

# xucheng at Xu-Chengs-MacBook-Pro in ~ on pyenv:3.3.0 [20:10:57]
$ python
Python 3.3.0 (default, Sep 19 2014, 20:09:04)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os;os.environ['__PYVENV_LAUNCHER__']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/xucheng/.pyenv/versions/3.3.0/lib/python3.3/os.py", line 669, in __getitem__
    value = self._data[self.encodekey(key)]
KeyError: b'__PYVENV_LAUNCHER__'
>>> import sys;sys.executable
'/Users/xucheng/.pyenv/versions/3.3.0/bin/python'
>>>

So there's a chance that this issue is python 3.4 specific.

[vsajip]

I wonder why the second and third results of your test have different python version with your first one.

You're right - I must have created the venvs from an older build. So I created some new venvs just now, and the results are different again!

vinay@iMac:~$ python3.3 /tmp/virtualenv.py --no-setuptools /tmp/venv33
Using base prefix '/usr/local'
New python executable in /tmp/venv33/bin/python3.3
Also creating executable in /tmp/venv33/bin/python
vinay@iMac:~$ python3.3 -m venv /tmp/venv33a
vinay@iMac:~$ /tmp/venv33/bin/python
Python 3.3.3+ (3.3:72560f9bb2a2, Jan 14 2014, 12:36:39) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.executable, sorted(os.environ)
('/tmp/venv33/bin/python', ['Apple_PubSub_Socket_Render', 'COMMAND_MODE', 'DISPLAY', 'GPG_AGENT_INFO', 'HOME', 'LANG', 'LOGNAME', 'MANPATH', 'PATH', 'PS1', 'PWD', 'SECURITYSESSIONID', 'SHELL', 'SHLVL', 'SSH_AUTH_SOCK', 'TERM', 'TERM_PROGRAM', 'TERM_PROGRAM_VERSION', 'TMPDIR', 'USER', 'WORKON_HOME', '_', '__CF_USER_TEXT_ENCODING'])
>>> ^D
vinay@iMac:~$ /tmp/venv33a/bin/python
Python 3.3.3+ (3.3:72560f9bb2a2, Jan 14 2014, 12:36:39) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.executable, sorted(os.environ)
('/tmp/venv33a/bin/python', ['Apple_PubSub_Socket_Render', 'COMMAND_MODE', 'DISPLAY', 'GPG_AGENT_INFO', 'HOME', 'LANG', 'LOGNAME', 'MANPATH', 'PATH', 'PS1', 'PWD', 'SECURITYSESSIONID', 'SHELL', 'SHLVL', 'SSH_AUTH_SOCK', 'TERM', 'TERM_PROGRAM', 'TERM_PROGRAM_VERSION', 'TMPDIR', 'USER', 'WORKON_HOME', '_', '__CF_USER_TEXT_ENCODING'])
>>> 

Now, neither of them has a __PYVENV_LAUNCHER__ key!

So, possibly some 3.3 versions are also affected.

[xu-cheng]

So the results got weird. __PYVENV_LAUNCHER__ shows in the place it shouldn't be and fails to show in the place it should be. But setting aside from this problem, I fail to see the reason why we need to use __PYVENV_LAUNCHER__ overwriting sys.executable in pip or distlib. As your test is shown, when using pyvenv, sys.executable would be sufficient.

[vsajip]

As your test is shown, when using pyvenv, sys.executable would be sufficient

Believe me, I didn't add this little complication for fun :-)

I think it may depend on exactly how Python is built. When I implemented venv, I had to modify the stub launcher on OS X so that the stub value was used. I'm pretty sure that sys.executable gave me a framework build location and not a simple path such as /usr/local/bin/python. This meant that we couldn't find pyvenv.cfg in a venv (as sys.executable didn't point into a venv), which led me to change the stub launcher to set the environment variable, and add code in site.py to look for that variable.

I don't claim to be a Python-on-OS X expert, and I'm not sure if anything has changed in the way Python is built on OS X or whether Homebrew builds differ from python.org builds. It's unfortunate that I can't build or install 3.4 on OS X to do further investigation.

What happens with a non-Homebrew Python 3.4 (e.g. from a python.org installer)?

[xu-cheng]

I just tested python 3.4.0 and 3.4.1 installed by pyenv. And it's surprise that neither of them have __PYVENV_LAUNCHER__ key. So now it's back to homebrew problem. I don't know why homebrew builds differ from others, especially after I reviewed its building method. One possible reason is that homebrew builds have an extra --enable-framework argument during compiling. Does @tdsmith have any opinion?

[vsajip]

FYI: In the 3.4 sources, __PYVENV_LAUNCHER__ occurs in the following places:

./Lib/site.py
454:    if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
455:        executable = os.environ['__PYVENV_LAUNCHER__']

./Lib/test/test_venv.py
52:        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in os.environ:
53:            executable = os.environ['__PYVENV_LAUNCHER__']
100:        if sys.platform == 'darwin' and ('__PYVENV_LAUNCHER__'
102:            executable =  os.environ['__PYVENV_LAUNCHER__']

./Lib/venv/__init__.py
120:        if sys.platform == 'darwin' and '__PYVENV_LAUNCHER__' in env:
121:            executable = os.environ['__PYVENV_LAUNCHER__']

./Mac/Tools/pythonw.c
199:        setenv("__PYVENV_LAUNCHER__", real_path, 1);

./Modules/main.c
628:        char* pyvenv_launcher = getenv("__PYVENV_LAUNCHER__");
637:                Py_FatalError("Cannot decode __PYVENV_LAUNCHER__");

Note that it is only set in one place: the stub launcher in Mac/Tools/pythonw.c. In all the others, it's just read.

IIUC, the stub launcher is only used in framework builds anyway, to allow GUI apps to work correctly. I'm not sure if the pyenv builds you mention are such. Here's an example using a Python 3.2 framework build:

vinay@iMac:~$ which python3.2
/usr/local/bin/python3.2
vinay@iMac:~$ python3.2
Python 3.2.3+ (3.2:0786dfc3b2b4, Dec 31 2012, 14:23:03) 
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; sys.executable
'/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python'

However, what is /usr/local/bin/python3.2? It's a symlink to the stub launcher for that framework build:

vinay@iMac:~$ ls -l /usr/local/bin/python3.2
lrwxr-xr-x  1 root  wheel  63 31 Dec  2012 /usr/local/bin/python3.2 -> /Library/Frameworks/Python.framework/Versions/3.2/bin/python3.2

So, the stub is at
/Library/Frameworks/Python.framework/Versions/3.2/bin/python3.2
whereas the real interpreter is at
/Library/Frameworks/Python.framework/Versions/3.2/Resources/Python.app/Contents/MacOS/Python.

[xu-cheng]

Now it makes much more sense. The pyenv builds don't have framework enable, while homebrew bulids have.

For homebrewed python, both /usr/local/bin/python3 and /usr/local/opt/python3/bin/python3 are ultimately pointed to the stub launcher /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3. Therefore, a __PYVENV_LAUNCHER__ key always exists.

If I use /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/Resources/Python.app/Contents/MacOS/Python as interpreter, the __PYVENV_LAUNCHER__ key is gone.

So now the problem is how to make __PYVENV_LAUNCHER__ behave correctly. What we need is that __PYVENV_LAUNCHER__ exists only when pyvenv is invoked. But now __PYVENV_LAUNCHER__ will always exist if we use the stub launcher. Am I correct?

[vsajip]

But now PYVENV_LAUNCHER will always exist if we use the stub launcher. Am I correct?

Yes, but that is correct behaviour and needed for correct operation in venvs created by pyvenv on OS X. I would assume that the python.org builds are framework builds - can you check what the behaviour with them is (in terms of whether pip3 does the right thing)?

When distlib writes shebangs for scripts installed into venvs, it must use os,environ[__PYVENV_LAUNCHER__] on OS X if available, otherwise the scripts won't work correctly (using the venv's interpreter, and with the venv in sys.path). It's not clear from your original post whether your problem relates to installing scripts into venvs, or something else. Can you clarify?

[xu-cheng]

My current problem has no concern with venvs. It's that homebrew uses stub launcher, therefore making sys.executable useless as it's overwritten by os.environ[__PYVENV_LAUNCHER__].

But I think this problem would be spread to any framework enabled builds. I don't know whether I understand correctly. Shouldn't the correct code look similar like this:

def get_executable():
    if sys.platform == 'darwin' and \
      ('__PYVENV_LAUNCHER__' in os.environ) and \
       venv_is_truly_running():
        result =  os.environ['__PYVENV_LAUNCHER__']
    else:
        result = sys.executable
    return result

[vsajip]

The code that is determining the shebang to use is running in an installer (like pip, but not necessarily pip). There is no reason to assume that the installer code is running in the venv being installed into, and it might not be.

The actual logic that distlib uses to determine the shebang to write is here. Here's an excerpt:

        if self.executable:
            executable = self.executable
            enquote = False     # assume this will be taken care of
        elif not sysconfig.is_python_build():
            executable = get_executable()
        elif in_venv():
            executable = os.path.join(sysconfig.get_path('scripts'),
                            'python%s' % sysconfig.get_config_var('EXE'))
        else:
            executable = os.path.join(
                sysconfig.get_config_var('BINDIR'),
               'python%s%s' % (sysconfig.get_config_var('VERSION'),
                               sysconfig.get_config_var('EXE')))

So, note that get_executable(), which you refer to in your original post, is only called if running from a source build, but not if running an installed Python. So it looks as though you are building the Homebrew Python from source and running that build, rather than installing it and running that - is that the case?

[vsajip]

Whoops, I think I misread the code I mentioned above (read it after coming in from a late night :-) Let me get some coffee and look at it again...

[vsajip]

Okay, I've had some coffee now, let's see if I can make a little more sense.

1. The shebangs written by distlib (and hence pip) refer to a stub launcher. This is important for venvs (so that the sys.path can be set up correctly for venvs) but also correct in the non-venv case, since running the stub launcher is essentially equivalent to running the corresponding Python interpreter.
2. You are seeing shebang executable paths like /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4 rather than /usr/local/opt/python3/bin/python3.4, which is bad because the former specifies a too-specific location and a minor version. However, both of these are paths pointing to stub launchers. It would seem that one is a symlink to the other (please confirm if that's not the case). If the __PYVENV_LAUNCHER__ environment variable is wrong, there's been an unwanted symlink resolution somewhere. But where could it be?
3. If os.environ[__PYVENV_LAUNCHER__] is pointing to the wrong path, that implies either a problem in the stub launcher code which sets the variable, or else that the too-specific path is used to invoke the launcher. When the stub launcher runs, this C code sets up __PYVENV_LAUNCHER__. Note that the comment says that realpath isn't called specifically to avoid losing symlink information.

ISTM it's worth checking why the environment variable is being set wrongly, and that means looking at the stub launcher code or the launcher invocation, rather than distlib or pip.

[xu-cheng]

Yeah you're right. So here're the test I run to determine different executes' realpath and os.environ['__PYVENV_LAUNCHER__']:

interpreter	realpath	os.environ['__PYVENV_LAUNCHER__']
/usr/local/bin/python3.4	../Cellar/python3/3.4.1_1/bin/python3.4	/usr/local/bin/python3.4
/usr/local/opt/python3/bin/python3.4	/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4	/usr/local/Cellar/python3/3.4.1_1/bin/python3.4
/usr/local/Cellar/python3/3.4.1_1/bin/python3.4	/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4	/usr/local/Cellar/python3/3.4.1_1/bin/python3.4
/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4	N/A	/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4

[xu-cheng]

In addition, the pip3 of homebrewed python is installed by command: /usr/local/Cellar/python3/3.4.1_1/bin/python3 -m ensurepip --upgrade. Which results pip3 has the shebang: #!/usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/bin/python3.4. Then the pip3's shebang will spread to any other scripts installed by it.

[vsajip]

So, if instead the ensurepip command were invoked using e.g. /usr/local/opt/python3/bin/python3, all would perhaps be well. Can you confirm that's the case?

[xu-cheng]

Test shows it's not helping. Still wrong shebang. But I don't think we should solve the issue by finding right invoked command. From my understanding, the desired behavior is like this:

• If in venv, using os.environ['__PYVENV_LAUNCHER__']
• Then using sys.executable if available.
• Fallback to OS specific default.

So which command is invoked shouldn't be the issue.

[vsajip]

The behaviour in distlib seems correct from the point of view of both operation in a venv and otherwise. Remember what I said: when writing shebangs for a script being installed into a venv, you don't need to have the venv active, nor use the venv's interpreter for that writing operation.

If python -m ensurepip is called with the correct symlink, the shebangs should be as expected - why is that not the correct solution? What exactly happens if you remove pip and reinstall it using one of the following commands?

/usr/local/opt/python3/bin/python3 -m ensurepip --upgrade

/usr/local/bin/python3 -m ensurepip --upgrade

Also, in get_executable(), what is the value of sys.executable? Is it a stub launcher, or not? Whatever it is, where did that value come from?

[xu-cheng]

As the test shown, invoking /usr/local/opt/python3/bin/python3 -m ensurepip --upgrade will result out the shebang as #!/usr/local/Cellar/python3/3.4.1_1/bin/python3. And invoking /usr/local/bin/python3 -m ensurepip --upgrade, the shebang will become #!/usr/local/bin/python3. Neither of them are correct.

And sys.executable is set by the file /usr/local/Cellar/python3/3.4.1_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/sitecustomize.py with the content of this. For more homebrew related information, @tdsmith should have better insight.

[vsajip]

Why is /usr/local/bin/python3 the wrong shebang? If it's too general, and you want to be 3.4-specific, then I presume there is a /usr/local/bin/python3.4 which produces analogous results.

Since I can't run Homebrew Python on my system, I was hoping you could e.g. set a break point in get_executable() [where you say the wrong value is being set] and see what the alternative value (sys.executable) is at that point.

[tdsmith]

I added a call to pdb.set_trace() inside get_executable() in /usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/utils.py.

darkside:distlib tim$ python3 -m pdb `which pip3` install -U --force-reinstall ipython
> /usr/local/bin/pip3(4)<module>()
-> import re
(Pdb) c
Downloading/unpacking ipython
  Downloading ipython-2.2.0-py3-none-any.whl (2.8MB): 2.8MB downloaded
Downloading/unpacking gnureadline (from ipython)
  Downloading gnureadline-6.3.3-cp34-cp34m-macosx_10_9_x86_64.whl (133kB): 133kB downloaded
Installing collected packages: ipython, gnureadline
  Found existing installation: ipython 2.2.0
    Uninstalling ipython:
      Successfully uninstalled ipython
> /usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/util.py(158)get_executable()
-> if sys.platform == 'darwin' and ('__PYVENV_LAUNCHER__'
(Pdb) sys.executable
'/usr/local/opt/python3/bin/python3.4'

[tdsmith]

Oops, that was dumb, sorry; the same is true without the extra "python3 -m pdb" wrapper.

darkside:site-packages tim$ pip3 install -U --force-reinstall ipython
...
    Uninstalling ipython:
      Successfully uninstalled ipython
> /usr/local/lib/python3.4/site-packages/pip/_vendor/distlib/util.py(158)get_executable()
-> if sys.platform == 'darwin' and ('__PYVENV_LAUNCHER__'
(Pdb) sys.executable
'/usr/local/opt/python3/bin/python3.4'

[tdsmith]

Note that the comment says that realpath isn't called specifically to avoid losing symlink information.

But then it says "It is nice to have the directory name as a cleaned up absolute path though, therefore call realpath on dirname(path)." Isn't that what's hosing us?

[vsajip]

Isn't that what's hosing us?

Perhaps. If so, it would need to be a change in Python itself. Why is this not an issue for stock (python.org) Python, which I believe is also a framework build? Or is it also an issue there? No issue has been reported on the stock Python builds, AFAIK.

[xu-cheng]

@vsajip The reason that I think /usr/local/bin/python3 is wrong shebang is because we cannot guarantee it's homebrewed python. There's a slight chance for some users to install other python builds. So it's best that we have a shebang like this /usr/local/opt/python3/bin/python3.4. And that's the weird place as /usr/local/opt/python3/bin/python3 -m ensurepip --upgrade resulting to #!/usr/local/Cellar/python3/3.4.1_1/bin/python3.

As for your unable testing homebrew on your Leopard, can you try test with tigerbrew?

EDIT sorry for copy/paste error.

[vsajip]

we cannot guarantee it's homebrewed python

OK, I see.

Unfortunately, my machine is a bit temperamental - although it has MacPorts installed and a whole bunch of MacPorts libraries, I still regularly get problems trying to install or upgrade stuff. So I'm wary of making any changes which might break something I need :-(

Possibly @tdsmith has identified where the symlink resolution occurs, but that would mean a change to Python itself (not distlib or pip), and before raising a Python issue, the question would need to be answered as to whether the python.org framework builds have the same issue, or not. (Although I have built Python on this machine earlier this year, I can't get it to build now because I can't upgrade libintl on my machine, and newer Python needs that.)

[tdsmith]

I can set up the python.org build but what is it we'd like to test? Homebrew doesn't patch Python and this manifestation is particular to the way Homebrew distributes python.

Applying this patch against pythonw.c during the Homebrew build does indeed get rid of the symlink resolution so that pip takes a shebang that matches the ensurepip invocation and writes shebangs that match its own: https://gist.github.com/tdsmith/f6f3ab95101baf5de5fc

I guess our next step is to open an issue against Python to see if we can convince them to change this behavior? I really appreciate your patience; this has been very helpful.

[vsajip]

While it was I that implemented the relevant change in pythonw.c, I don't profess to be an expert on framework builds. If you open a Python issue, I would hope that the Mac experts on python-dev can pronounce on it. I have no objection to your patch, but I believe (can't quite remember) that the original change was okayed by others. I would suggest adding me and Ronald Oussoren or Ned Deily to the nosy list for the issue.

[ned-deily]

I'm not a homebrew user so I don't have a setup to test this at the moment. But, if I understand from the description above, everything works correctly in the non-wheel case, (which I assume means that Distutils build_scripts.py is doing the shebang modification?) and not in the wheel case when distlib is used to do it. So, before we think about changing the stub launcher, it seems to me a first issue to resolve is why does the behavior of distlib differs from that of Distutils, which is after all the standard?

[dstufft]

It's likely to actually be setuptools doing it. Probably setuptools just uses sys.executable since it was written before __PYVENV_LAUNCHER__ was a thing.

[dstufft]

Added this to the 6.0 milestone and I've emailed @vsajip asking for a new release of distlib. Hopefully that comes out in time for 6.0 but if not I think we can work around it.