test_test_builder.py not finding a suitable interpreter on OS X

[benjyw]

Reproduces (at least on some OS X systems) with:

PANTS_DEV=1 ./pants goal test tests/python/pants_test/backend/python:test_builder

The problem manifests here.

sys.executable is: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

And the interpreters available in the interpreter cache are:

/Users/benjy/src/pants/build-support/pants_dev_deps.venv/bin/python
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7

So there's no string match.

However note that, although you might reasonably think that the second interpreter in the cache is just a symlink to sys.executable, they are actually different inodes of files with different md5sums.

[jsirois]

It might be useful to know what os.path.realpath(sys.prefix) says for each of /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python and /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 - hopefully they give the same answer since the lib dir is a dir.

[benjyw]

>>> os.path.realpath(sys.prefix)
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7'
>>> os.path.realpath('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python')
'/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python'
>>> os.path.realpath('/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7')
'/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7'

[jsirois]

Sorry - to be clear, these 2 should be equal:

/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -c 'import os, sys; print(os.path.realpath(sys.prefix))'
/System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 -c 'import os, sys; print(os.path.realpath(sys.prefix))'

[benjyw]

Correct, these both yield:

/System/Library/Frameworks/Python.framework/Versions/2.7

[wickman]

Interesting -- so two separate binaries but tied to the same Python standard library installation. I wonder if we need to make pex.interpreter aware of sys.prefix and then somehow tie that into the interpreter cache.

[jsirois]

I think so. I never checked content hashes for java/javac, but when Apple packaged java 1.6 it did this same general packaging thing. It had a mac app-in-a-dir style Contents and then the java standard layout and these 2 layouts were maintained via symlinks and - I thought - binary hardlinks, although perhaps the Contents binaries were - as here - slightly different.

[benjyw]

Any news on this? Not being able to run a CI locally is getting to be a hassle...

[jsirois]

Note that this should be easy to workaround on a case by case basis. Install your own python2.7 whether by compiling the distribution tarball yourself, using https://github.com/yyuu/pyenv or brew or what have you and placing this at the head of your PATH.

[benjyw]

That has always been my state:

fozzie:[~/src/pants][master]$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/share/npm/bin
fozzie:[~/src/pants][master]$ which python | xargs ls -l
lrwxr-xr-x  1 benjy  admin  33 Mar 17  2014 /usr/local/bin/python -> ../Cellar/python/2.7.6/bin/python
fozzie:[~/src/pants][master]$ which python2 | xargs ls -l
lrwxr-xr-x  1 benjy  admin  34 Mar 17  2014 /usr/local/bin/python2 -> ../Cellar/python/2.7.6/bin/python2
fozzie:[~/src/pants][master]$ which python2.7 | xargs ls -l
lrwxr-xr-x  1 benjy  admin  36 Mar 17  2014 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.6/bin/python2.7

As far as it's up to me, I only use brew python.

[jsirois]

This makes little sense to me then. Your sys.executable should be a brew one then which should have a sane binary path to do comparisons against.

[benjyw]

My pants binary does indeed run on the brew python. However PythonTask.select_interpreter() is choosing the system one to actually run the tests on. Investigating further.

[benjyw]

This is because PythonInterpreterCache.select_interpreter() returns the interpreter with min(compatibilities), which is the system 2.7.5, not the brew 2.7.6.

So pytest executes tests on the system 2.7.5, which is what test_test_builder.py then sees.

[mateor]

I have been hitting this myself. I didn't realize it was a larger issue and had been running the CI on a totally separate box. All tests pass until the interpreter cache near the end.

My OSX machine is getting its backlight repaired but I will report my state early next week.

[jsirois]

Re-discovered in #1908 if I'm not mistaken.

I think this was solved when we upgraded to pex 1.0.3 which has this fix for pex-tool/pex#144 for OSX 'Python' (vs 'python') which would previously lead to missed interpreters.