test_setup_using_eggs failing locally on my laptop

[ericzundel]

This is with pants at 0.0.40 on my macbook pro with python 2.7.9 (installed from brew)

$./pants test tests/python/pants_test:all
...
                     ==================== FAILURES ====================
                     ___ TestInterpreterCache.test_setup_using_eggs ___

                     self = <pants_test.python.test_interpreter_cache.TestInterpreterCache testMethod=test_setup_using_eggs>

                         def test_setup_using_eggs(self):
                           def options(**kwargs):
                             return create_option_values(kwargs)

                           def link_egg(repo_root, requirement):
                             existing_dist_location = self._interpreter.get_location(requirement)
                             if existing_dist_location is not None:
                               existing_dist = Package.from_href(existing_dist_location)
                               requirement = '{}=={}'.format(existing_dist.name, existing_dist.raw_version)

                             distributions = resolve([requirement],
                                                     interpreter=self._interpreter,
                                                     precedence=(EggPackage, SourcePackage))
                             self.assertEqual(1, len(distributions))
                             dist_location = distributions[0].location

                             self.assertRegexpMatches(dist_location, r'\.egg$')
                             os.symlink(dist_location, os.path.join(repo_root, os.path.basename(dist_location)))

                             return Package.from_href(dist_location).raw_version

                           with temporary_dir() as root:
                             egg_dir = os.path.join(root, 'eggs')
                             os.makedirs(egg_dir)
                             setuptools_version = link_egg(egg_dir, 'setuptools')
                             wheel_version = link_egg(egg_dir, 'wheel')

                             python_setup_options = options(interpreter_cache_dir=None,
                                                            pants_workdir=os.path.join(root, 'workdir'),
                                                            interpreter_requirement=self._interpreter.identity.requirement,
                                                            setuptools_version=setuptools_version,
                                                            wheel_version=wheel_version)
                             python_setup = PythonSetup('test-scope', python_setup_options)
                             python_repos = PythonRepos('test-scope', options(indexes=[], repos=[egg_dir]))
                             cache = PythonInterpreterCache(python_setup, python_repos)

                             interpereters = cache.setup(paths=[os.path.dirname(self._interpreter.binary)],
                                                         filters=[str(self._interpreter.identity.requirement)])
                     >       self.assertGreater(len(interpereters), 0)
                     E       AssertionError: 0 not greater than 0

                     tests/python/pants_test/python/test_interpreter_cache.py:100: AssertionError
                      1 failed, 1099 passed, 7 xfailed, 1 xpassed, 3 warnings in 363.85 seconds 

FAILURE

[jsirois]

I'll need your help debugging this. For a start, this may given more info:
PEX_VERBOSE=5 ./pants clean-all test tests/python/pants_test/python:test_interpreter_cache -- -sktest_setup_using_eggs.

The line before the failing assert is just taking the current interpreter pants is running under and using its dirname as the path and its identity, ie: CPython-2.7.9 as the filter in an attempt to get back the same interpreter, but setup with setuptools and wheel eggs installed. Of those ingredients, the most likely miss is the setuptools or wheel egg failing to install. The only other miss I can imagine is the dirname operation not playing well with symlinks in some way. This last is very handwavy and I'm having a hard time seeing how binary_path -> dirname -> find same binary at that path could fail.

[ericzundel]

Here's the full output of running the above command:

https://gist.github.com/ericzundel/e988d008ed31a3760027

Note that the test output reports its using 2.7.6, which is what is installed in /usr/bin on my mac.

                     ============== test session starts ===============
                     platform darwin -- Python 2.7.6 -- py-1.4.30 -- pytest-2.6.4
                     plugins: cov, timeout
                     collected 4 items 

The version of python on PATH is 2.7.9 though:

$ python -v 
...
dlopen("/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so", 2);
import readline # dynamically loaded from /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/readline.so

[jsirois]

OK, thanks. More questions:

1. what is your PATH

2. what do you get for each of these:

   a. /usr/bin/python --version
   b. /usr/bin/python2.7 --version

3. does either /usr/bin/python or which python have a symlinked element in any of its path components?

[ericzundel]

The Python I want is in /usr/local/bin and it is first in the path:

$ which python
/usr/local/bin/python

$ echo $PATH
/Users/zundel/.rvm/gems/ruby-2.2.0/bin:/Users/zundel/.rvm/gems/ruby-2.2.0@global/bin:/Users/zundel/.rvm/rubies/ruby-2.2.0/bin:/Users/zundel/Development/topsoil/bin:/usr/local/bin:/usr/local/sbin:/usr/local/share/npm/bin:/Users/zundel/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/usr/square/bin:/usr/local/sbin:/Users/zundel/Development/go/bin:/usr/local/go/bin:/usr/local/share/scala/bin:/Users/zundel/.rvm/bin

the versions installed in /usr/bin:

$ /usr/bin/python --version
Python 2.7.6

$ /usr/bin/python2.7 --version
Python 2.7.6

And the answer is "Yes" /usr/local/bin/python is a symlink (I think this is normal when Python is installed by brew)

~ ls -ld /usr
drwxr-xr-x@ 14 root  wheel  476 Jul  9 13:06 /usr
~ ls -ld /usr/
drwxr-xr-x@ 14 root  wheel  476 Jul  9 13:06 /usr/
~ ls -ld /usr/local/
drwxrwxr-x  28 root  admin  952 Jul 29 10:44 /usr/local/
~ ls -ld /usr/local/bin/
drwxr-xr-x  372 zundel  admin  12648 Aug  2 09:49 /usr/local/bin/
~ ls -ld /usr/local/bin/python
lrwxr-xr-x  1 zundel  admin  33 Mar  5 19:05 /usr/local/bin/python -> ../Cellar/python/2.7.9/bin/python

[jsirois]

Hrm. Do you mind moving aside and saving the complete build-support/ dir then re-trying?
I mainly interested in seeing if the pants bootstrap venv was built back when 2.7.6 was your default python and if so, if thats having a strange interaction with the fact 2.7.9 is now your default.

[jsirois]

By moving aside, I guess I mean tar/zip it up somewhere safe then git clean -fdx build-support/

[ericzundel]

I did a git clean -fdx and ran the command again and the test still fails: https://gist.github.com/ericzundel/879124f4e8bf7b8cf5e1

I upgraded to Yosemite (macos 10.10) about a month ago. When you say 'default' python 2.7.6 it is the version installed in /usr/bin/, I'm not sure if that changed from 10.9 to 10.0. However, I've had a version higher than 2.7.6 in /usr/local/bin for a long time. I've had a script in place to enforce a minimum runtime version of python for pants of 2.7.8 for our users since March 2015.

[jsirois]

OK - not even close to a solution, but noting some facts as I think through this:

1. your gist shows pytest is running under 2.7.6
2. pytest is run via a pex chroot and that chroot is built to use a compatible interpreter for all python_tests targets in-play. That means in this case, no particular extra compatibility constraints are in play and the lowest interpreter meeting the global constraint of CPython>=2.7,<3 is selected which is met by 2.7.6, see: https://github.com/jsirois/pants/blob/master/src/python/pants/backend/python/tasks/python_task.py#L92

Open questions here for myself:

1. Can we do better than lowest? This is almost certainly irrelevant to the solution to this particular issue though. If the "expected" interpreter were selected, the test would likely not error - but the test should not be erroring on the interpreter it got.
2. Given that pytest is in fact running under 2.7.6, does the layout of OSX 2.7.6 suggest a reason this test could fail to find/setup its current interpreter via the dirname(python_exe_path) if version == current_version strategy used in https://github.com/pantsbuild/pants/blob/master/tests/python/pants_test/python/test_interpreter_cache.py#L96-L97

[jsirois]

...ok - the 2.7.6 thing is hard to debug without an OSX, I'm basically going to have to ask you to start adding prints to figure this out remotely. Before doing that, I would be interested in you changing your pants.ini interpreter compatbility constraint if its not already set to >=2.7.8, ie;

[python-setup]
interpreter_requirement: CPython>=2.7.8,<3

With a clean-all I'm guessing this test might pass since that constraint should keep 2.7.6 out of the interpreter cache, thus the brew python should be selected, and its presumably more sane path structure (or whatever is leading to this bug), should allow the test to pass. This would not be a fix I'm satisfied with, but it could shed more light.

[jsirois]

A pre-emptive debug request after the 2.7.8 constraint test is to apply this patch and report back on the results (w/o the 2.7.8 forcing bit in-play):

$ git diff
diff --git a/tests/python/pants_test/python/test_interpreter_cache.py b/tests/python/pants_test/python/test_interpreter_cache.py
index f48d229..0cffee6 100644
--- a/tests/python/pants_test/python/test_interpreter_cache.py
+++ b/tests/python/pants_test/python/test_interpreter_cache.py
@@ -93,6 +93,13 @@ class TestInterpreterCache(unittest.TestCase):
       python_repos = create_subsystem(PythonRepos, indexes=[], repos=[egg_dir])
       cache = PythonInterpreterCache(python_setup, python_repos)

+      print('')
+      print('>>> interpreter selection inputs')
+      print('    self._interpreter.binary: {}'.format(self._interpreter.binary))
+      print('    path: {}'.format(os.path.dirname(self._interpreter.binary)))
+      print('    filter: {}'.format(interpreter_requirement))
+      print('')
+
       interpereters = cache.setup(paths=[os.path.dirname(self._interpreter.binary)],
                                   filters=[str(interpreter_requirement)])
       self.assertGreater(len(interpereters), 0)

[ericzundel]

The 2.7.8 constraint test worked (the test passed). Output here: https://gist.github.com/ericzundel/201a88dc370e4b87482c

Here's the output with the diff above applied (and the test failing): https://gist.github.com/ericzundel/169b0a1d5045946d957e

[jsirois]

Thank you very much, that solves it. The binary is Python and the regexes pex uses assume python: https://github.com/pantsbuild/pex/blob/master/pex/interpreter.py#L181-L182

I'll fork off an issue and PR for pex.

[jsirois]

Fix is out here: https://rbcommons.com/s/twitter/r/2586/

[ericzundel]

I made that change to pex locally but I'm not sure how to point pants at it to see if this solves the issue on my laptop.

[jsirois]

Its easiest to just edit build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/interpreter.py and then to reverse, git clean -fdx build-support/ after.

[ericzundel]

Unfortunately, the problem persists after that change.

[ericzundel]

In the debugging information you added above:

>>> interpreter selection inputs
                         self._interpreter.binary: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
                         path: /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS
                         filter: CPython==2.7.6

Python with a capital P is the wrong one

$ /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python --version
Python 2.7.6

The python I actually want is named 'python2.7' or 'python'

$ ls -l /usr/local/bin/python2.7
lrwxr-xr-x  1 zundel  admin  36 Mar  5 19:05 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.9/bin/python2.7
$ ls -l /usr/local/Cellar/python/2.7.9/bin/python2.7
lrwxr-xr-x  1 zundel  admin  85 Feb  9 22:30 /usr/local/Cellar/python/2.7.9/bin/python2.7 -> /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/bin/python2.7
$ /usr/local/bin/python2.7 --version
Python 2.7.9

$ ls -l /usr/local/bin/python
lrwxr-xr-x  1 zundel  admin  33 Mar  5 19:05 /usr/local/bin/python -> ../Cellar/python/2.7.9/bin/python
$  ls -l /usr/local/Cellar/python/2.7.9/bin/python
lrwxr-xr-x  1 zundel  admin  82 Feb  9 22:30 /usr/local/Cellar/python/2.7.9/bin/python -> /usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/bin/python
$  /usr/local/bin/python --version
Python 2.7.9

PATH on my machine has /usr/local/bin before /usr/bin

/Users/zundel/.rvm/gems/ruby-2.0.0-p353/bin:/Users/zundel/.rvm/gems/ruby-2.0.0-p353@global/bin:/Users/zundel/.rvm/rubies/ruby-2.0.0-p353/bin:/Users/zundel/Development/topsoil/bin:/usr/local/bin:/usr/local/sbin:/usr/local/share/npm/bin:/Users/zundel/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/munki:/usr/square/bin:/usr/local/sbin:/Users/zundel/Development/go/bin:/usr/local/go/bin:/usr/local/share/scala/bin:/Users/zundel/.rvm/bin

[jsirois]

I suspect my try it out quick advice was bad. The 'venv' in play is a pex chroot built for tests and that uses the pex in 3rdparty/python/requirements.txt.

So... longer story:

1. In pex clone with patch, edit pex/version.py to use new rev, then run python2.7 setup.py sdist - you should now have a new version of pex in dist/ (a tarball).
2. In pants, add the pex dist dir to the [python-setup] repos list as a file url (should end with slash).
3. In pants edit 3rdparty/python/requirements.txt to have an exact requirement for the pex you burned locally, ie: pex==9.9.9

The local edit to the pex venv source in pants should continue to be irrelevant.

[jsirois]

It may not have been clear from above, but pants picking 2.7.6 is not wrong given a compatibility of >=2.7, <3. Granted it's not what you want, but it should work. I'd like to focus solely on getting it working, ie: dealing with OSX funky python packaging as Python.

[jsirois]

Working the current logic for picking a compatible interpreter to respect PATH order instead of picking lowest compatible (the current logic) - is valid, but I'd like to fork an issue for that alone.

[ericzundel]

With this test setup I was able to show that the fix is working:

diff --git a/pex/version.py b/pex/version.py
index a44de64..92c2ab2 100644
--- a/pex/version.py
+++ b/pex/version.py
@@ -1,7 +1,7 @@
 # Copyright 2015 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

-__version__ = '1.0.2'
+__version__ = '1.0.2.dev'

 SETUPTOOLS_REQUIREMENT = 'setuptools>=2.2,<16'
 WHEEL_REQUIREMENT = 'wheel>=0.24.0,<0.25.0'

Then run

$ python 2.7 setup.py sdist

and in Pants:

diff --git a/3rdparty/python/requirements.txt b/3rdparty/python/requirements.txt
index ca4c6c4..b29fe87 100644
--- a/3rdparty/python/requirements.txt
+++ b/3rdparty/python/requirements.txt
@@ -8,7 +8,9 @@ mox==0.5.3

 # Until https://github.com/pantsbuild/pex/issues/28 is addressed, we cannot
 # do pex<1.1.0.
-pex>=1.0.2,<1.0.999999
+#pex>=1.0.2,<1.0.999999
+-f /Users/zundel/Src/pex/dist/
+pex==1.0.2.dev0

 psutil==3.1.1
 lockfile==0.10.2
diff --git a/pants.ini b/pants.ini
index 5f77a7d..b6fbb79 100644
--- a/pants.ini
+++ b/pants.ini
@@ -167,6 +167,11 @@ restrict_push_urls: [
 # acceptable set should specify an explicit compatibility constraint.
 interpreter_requirement: CPython>=2.7,<3

+[python-repos]
+indexes: [
+    '/Users/zundel/Src/pex/dist',
+    'https://pypi.python.org/simple/'
+  ]

 [sign]
 # Default to debug keystore installed with SDK.

(Just setting python-repos was not enough. I tried a file:/// url as well)

[jsirois]

Excwllwnt. Thanks for working through this with me.

I typed the config, should have been the [python-repos] repos list. The indices must conform to cheeseshop index format which a local flat dist dir does not, aka: I'm pretty sure your indexes entry went completely unused by pex via silent failures to find an index. The -f in requirements.txt saved the day all on its own.

[jsirois]

Fix is in @ 65ac286

Please re-open if the test still fails for you.