PEX doesn't scrub "local" from virtualenv site-packages #21

kevints · 2014-10-22T23:10:48Z

In recent versions of virtualenv there's a "local" directory that just contains symlinks

% ./build-support/virtualenv --version
1.11.6

% ls -l build-support/pants.venv/local
total 12
lrwxrwxrwx 1 ksweeney ksweeney 60 Oct 22 16:03 bin -> /home/ksweeney/workspace/aurora/build-support/pants.venv/bin
lrwxrwxrwx 1 ksweeney ksweeney 64 Oct 22 16:03 include -> /home/ksweeney/workspace/aurora/build-support/pants.venv/include
lrwxrwxrwx 1 ksweeney ksweeney 60 Oct 22 16:03 lib -> /home/ksweeney/workspace/aurora/build-support/pants.venv/lib

PEX doesn't scrub this local site-packages from sys.path, causing pants dependencies to shadow pex dependencies:

Minimal test case:

def test_psutil():
  import sys
  print('\n'.join(sys.path))
  import psutil
  assert psutil.__version__ == '2.1.1'

Test output:

/home/ksweeney/workspace/aurora/src/test/python/apache/aurora/executor
/tmp/tmpV7InVa/.bootstrap
/tmp/tmpV7InVa
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/plat-x86_64-linux-gnu
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-tk
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-old
/home/ksweeney/workspace/aurora/build-support/pants.venv/lib/python2.7/lib-dynload
/usr/lib/python2.7
/usr/lib/python2.7/plat-x86_64-linux-gnu
/usr/lib/python2.7/lib-tk
/home/ksweeney/workspace/aurora/build-support/pants.venv/local/lib/python2.7/site-packages
/tmp/tmpV7InVa/.deps/unittest2-0.5.1-py2-none-any.whl
/tmp/tmpV7InVa/.deps/pytest_cov-1.8.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/pytest_timeout-0.4-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/pytest-2.6.3-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.string-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.quantity-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.python-0.5.4-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.metrics-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.exceptions-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/psutil-2.1.1-cp27-none-linux_x86_64.whl
/tmp/tmpV7InVa/.deps/twitter.common.contextutil-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/py-1.4.25-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/cov_core-1.14.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.lang-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/wheel-0.24.0-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/setuptools-7.0-py2.py3-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.decorators-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/twitter.common.dirutil-0.3.0-py2-none-any.whl
/tmp/tmpV7InVa/.deps/coverage-4.0a1-cp27-none-linux_x86_64.whl
F

============================================== FAILURES ==============================================
____________________________________________ test_psutil _____________________________________________

    def test_psutil():
      import sys
      print('\n'.join(sys.path))
      import psutil
>     assert psutil.__version__ == '2.1.1'
E     assert '1.1.3' == '2.1.1'
E       - 1.1.3
E       + 2.1.1

As a workaround, removing the "local" directory makes isolation work correctly:

~aurora git aurora/. kts/upgrade-virtualenv  U  
% rm -fr build-support/pants.venv/local                                     
~aurora git aurora/. kts/upgrade-virtualenv  U  
% ./pants src/test/python/apache/aurora/executor:executor_vars -s -k test_psutil
src.test.python.apache.aurora.executor.executor_vars                            .....   SUCCESS

The text was updated successfully, but these errors were encountered:

wickman · 2014-11-21T02:50:19Z

Could not repro:

mba=repro=; pex -r psutil -v -o psutil.pex
twitter.common.python.http: Crawling /Users/wickman/.pex/build: 0.2ms
twitter.common.python.http: Crawling /Users/wickman/.pex/build: 0.2ms
Resolved distributions:
  psutil 2.1.3
Creating environment PEX.
Saving PEX file to psutil.pex

mba=repro=; virtualenv --version
1.11.6

mba=repro=; virtualenv psutil
New python executable in psutil/bin/python2.7
Also creating executable in psutil/bin/python
Installing setuptools, pip...done.

mba=repro=; source psutil/bin/activate
(psutil)mba=repro=; pip install psutil==1.1.3
Downloading/unpacking psutil==1.1.3
  Downloading psutil-1.1.3.tar.gz (165kB): 165kB downloaded
  Running setup.py (path:/private/tmp/repro/psutil/build/psutil/setup.py) egg_info for package psutil

Installing collected packages: psutil
  Running setup.py install for psutil
    building '_psutil_osx' extension
    /opt/twitter/bin/gcc-4.2 -fno-strict-aliasing -I/tmp/python.IVpkzV/readline/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/wickman/Python/CPython-2.7.8/include/python2.7 -c psutil/_psutil_osx.c -o build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_osx.o
    /opt/twitter/bin/gcc-4.2 -fno-strict-aliasing -I/tmp/python.IVpkzV/readline/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/wickman/Python/CPython-2.7.8/include/python2.7 -c psutil/_psutil_common.c -o build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_common.o
    /opt/twitter/bin/gcc-4.2 -fno-strict-aliasing -I/tmp/python.IVpkzV/readline/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/wickman/Python/CPython-2.7.8/include/python2.7 -c psutil/arch/osx/process_info.c -o build/temp.macosx-10.4-x86_64-2.7/psutil/arch/osx/process_info.o
    /opt/twitter/bin/gcc-4.2 -bundle -undefined dynamic_lookup -L/tmp/python.IVpkzV/readline/lib build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_osx.o build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_common.o build/temp.macosx-10.4-x86_64-2.7/psutil/arch/osx/process_info.o -o build/lib.macosx-10.4-x86_64-2.7/_psutil_osx.so -framework CoreFoundation -framework IOKit
    ld: warning: directory not found for option '-L/tmp/python.IVpkzV/readline/lib'
    building '_psutil_posix' extension
    /opt/twitter/bin/gcc-4.2 -fno-strict-aliasing -I/tmp/python.IVpkzV/readline/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/wickman/Python/CPython-2.7.8/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_posix.o
    /opt/twitter/bin/gcc-4.2 -bundle -undefined dynamic_lookup -L/tmp/python.IVpkzV/readline/lib build/temp.macosx-10.4-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.macosx-10.4-x86_64-2.7/_psutil_posix.so
    ld: warning: directory not found for option '-L/tmp/python.IVpkzV/readline/lib'

Successfully installed psutil
Cleaning up...

(psutil)mba=repro=; ./psutil.pex 
Python 2.7.8 (default, Nov  3 2014, 10:09:47) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import psutil
>>> psutil.__version__
'2.1.3'

wickman · 2014-11-21T02:51:27Z

Could you run with PEX_VERBOSE=5 to provide some information about environment scrubbing? When I do so, it correctly identifies the scrubbed elements:

(psutil)mba=repro=; PEX_VERBOSE=5 ./psutil.pex
twitter.common.python.pex: Found site-library: /private/tmp/repro/psutil/lib/python2.7/site-packages
twitter.common.python.pex: Inspecting path element: /private/tmp/repro/psutil/lib/python2.7/site-packages
twitter.common.python.pex: Scrubbing from site-packages: /private/tmp/repro/psutil/lib/python2.7/site-packages
twitter.common.python.environment: Activating PEX virtual environment :: Searching dependency cache: ./psutil.pex/.deps :: Caching psutil 2.1.3 into /Users/wickman/.pex/install/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl.005f999c29eed10307ca98twitter.common.python.environment: Activating PEX virtual environment :: Searching dependency cache: ./psutil.pex/.deps :: Adding psutil 2.1.3:/Users/wickman/.pex/install/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl.005f999c29eed10307ca98f788ectwitter.common.python.environment: Activating PEX virtual environment :: Resolving psutil==2.1.3                                                                                                                                                     twitter.common.python.environment: Activating PEX virtual environment: 13.9ms                                     
twitter.common.python.environment:   Searching dependency cache: ./psutil.pex/.deps: 7.0ms
twitter.common.python.environment:     Caching psutil 2.1.3 into /Users/wickman/.pex/install/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl.005f999c29eed10307ca98f788ec09a10157f899: 0.4ms
twitter.common.python.environment:     Adding psutil 2.1.3:/Users/wickman/.pex/install/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl.005f999c29eed10307ca98f788ec09a10157f899/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl: 0.2ms
twitter.common.python.environment:   Resolving psutil==2.1.3: 6.4ms
twitter.common.python.environment:   Activating psutil 2.1.3: 0.4ms
twitter.common.python.environment:     Adding sitedir: 0.2ms
twitter.common.python.pex: PYTHONPATH contains:
twitter.common.python.pex:   * /private/tmp/repro/psutil.pex/.bootstrap
twitter.common.python.pex:     /private/tmp/repro/psutil.pex
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python27.zip
twitter.common.python.pex:     /private/tmp/repro/psutil/lib/python2.7
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python2.7/plat-darwin
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python2.7/plat-mac
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python2.7/plat-mac/lib-scriptpackages
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python2.7/lib-tk
twitter.common.python.pex:   * /private/tmp/repro/psutil/lib/python2.7/lib-old
twitter.common.python.pex:     /private/tmp/repro/psutil/lib/python2.7/lib-dynload
twitter.common.python.pex:     /Users/wickman/Python/CPython-2.7.8/lib/python2.7
twitter.common.python.pex:     /Users/wickman/Python/CPython-2.7.8/lib/python2.7/plat-darwin
twitter.common.python.pex:     /Users/wickman/Python/CPython-2.7.8/lib/python2.7/lib-tk
twitter.common.python.pex:     /Users/wickman/Python/CPython-2.7.8/lib/python2.7/plat-mac
twitter.common.python.pex:     /Users/wickman/Python/CPython-2.7.8/lib/python2.7/plat-mac/lib-scriptpackages
twitter.common.python.pex:     /Users/wickman/.pex/install/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl.005f999c29eed10307ca98f788ec09a10157f899/psutil-2.1.3-cp27-none-macosx_10_4_x86_64.whl
twitter.common.python.pex:   * - paths that do not exist or will be imported via zipimport
twitter.common.python.pex: No entry point specified, dropping into interpreter

wickman · 2014-11-21T02:54:15Z

I cloned the aurora incubator repository and my pants.venv did not contain a 'local' directory. Perhaps this is a Linux vs Mac thing? I'd love to know more about your specific environment to learn how to reproduce.

wickman · 2014-11-21T02:58:31Z

perhaps it could be related to this bit of code in virtualenv.py:

def fix_local_scheme(home_dir, symlink=True):
    """
    Platforms that use the "posix_local" install scheme (like Ubuntu with
    Python 2.7) need to be given an additional "local" location, sigh.
    """
    try:
        import sysconfig
    except ImportError:
        pass
    else:
        if sysconfig._get_default_scheme() == 'posix_local':
            local_path = os.path.join(home_dir, 'local')
            if not os.path.exists(local_path):
                os.mkdir(local_path)
                for subdir_name in os.listdir(home_dir):
                    if subdir_name == 'local':
                        continue
                    copyfile(os.path.abspath(os.path.join(home_dir, subdir_name)), \
                                                            os.path.join(local_path, subdir_name), symlink)

wickman · 2014-11-21T03:19:00Z

OK, I found a Linux box and was able to repro this. Investigating a possible fix.

wickman · 2014-11-21T04:09:52Z

Fix forthcoming. A sprinkling of os.path.realpath here and there, y'know.

wickman · 2014-11-21T18:17:34Z

https://rbcommons.com/s/twitter/r/1379/

SSIA. The user facing parts of this change: - package resolution happens via requests now -- this is way more reliable and results in fewer Untranslateable exceptions - PEX_VERBOSE now controls all pex verbosity, including better messages around Untranslateable (yay!) - fixes a major pex regression with namespace packages introduced somewhere in 0.5.x The pex 0.7.0 -> 0.8.0 changelog: * *API change*: Decouple translation from package iteration. This removes the Obtainer construct entirely, which likely means if you're using PEX as a library, you will need to change your code if you were doing anything nontrivial. This adds a couple new options to ``resolve`` but simplifies the story around how to cache packages. [RB #785](https://rbcommons.com/s/twitter/r/785/) * Refactor http handling in pex to allow for alternate http implementations. Adds support for [requests](https://github.com/kennethreitz/requests), improving both performance and security. For more information, read the commit notes at [91c7f32](pex-tool/pex@91c7f32). [RB #778](https://rbcommons.com/s/twitter/r/778/) * Improvements to API documentation throughout. * Renamed ``Tracer`` to ``TraceLogger`` to prevent nondeterministic isort ordering. * Refactor tox.ini to increase the number of environment combinations and improve coverage. * Adds HTTP retry support for the RequestsContext. [RB #1303](https://rbcommons.com/s/twitter/r/1303/) * Make pex --version correct. [Issue #19](pex-tool/pex#19) * Bug fix: Fix over-aggressive sys.modules scrubbing for namespace packages. Under certain circumstances, namespace packages in site-packages could conflict with packages within a PEX, causing them to fail importing. [RB #1378](https://rbcommons.com/s/twitter/r/1378/) * Bug fix: Replace uses of ``os.unsetenv(...)`` with ``del os.environ[...]`` [Pull Request #11](pex-tool/pex#11) * Bug fix: Scrub sys.path and sys.modules based upon both supplied path and realpath of files and directories. Newer versions of virtualenv on Linux symlink site-packages which caused those packages to not be removed from sys.path correctly. [Issue #21](pex-tool/pex#21) * Bug fix: The pex -s option was not correctly pulling in transitive dependencies. [Issue #22](pex-tool/pex#22) * Bug fix: Adds ``content`` method to HTTP contexts that does HTML content decoding, fixing an encoding issue only experienced when using Python 3. [Issue #10](pex-tool/pex#10) Testing Done: build-support/bin/ci.sh Reviewed at https://rbcommons.com/s/twitter/r/1421/

wickman self-assigned this Nov 21, 2014

wickman closed this as completed in eeb80a5 Nov 21, 2014

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PEX doesn't scrub "local" from virtualenv site-packages #21

PEX doesn't scrub "local" from virtualenv site-packages #21

kevints commented Oct 22, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

PEX doesn't scrub "local" from virtualenv site-packages #21

PEX doesn't scrub "local" from virtualenv site-packages #21

Comments

kevints commented Oct 22, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014

wickman commented Nov 21, 2014