sys.path rewrite in Python 2 regression

[jd]

Version >= 20.0.0 of virtualenv changed the way sys.path is modified by the site.py file (at least for Linux). That breaks at least one of our test cases.

When PYTHONPATH is set to a directory that is inside virtualenv and starts with sys.prefix it's rewritten to target the sys.base_prefix.

The problem is that is the directory might not be part of the standard library nor install in pyenv. This is our case since we're using tox to install our library in the virtualenv and then setting PYTHONPATH to a subdirectory somewhere in this installed package.

IIRC the culprit
https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/via_global_ref/builtin/python2/site.py#L96-L97

[jd]

Here's how to reproduce:

➔ python -m virtualenv venv2006
created virtual environment CPython2.7.17.final.0-64 in 402ms
  creator CPython2macOsFramework(dest=/Users/jd/venv2006, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/jd/Library/Application Support/virtualenv/seed-app-data/v1)
  activators PythonActivator,CShellActivator,FishActivator,PowerShellActivator,BashActivator
➔ export PYTHONPATH=$PWD/venv2006/foobar
➔ venv2006/bin/python -c 'import sys, pprint;pprint.pprint(sys.path)'
['',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/foobar',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/jd/venv2006/lib/python2.7/site-packages']

(reproducible on macOS)

[gaborbernat]

An oversight on our side, we should skip PYTHONPATH paths here https://github.com/pypa/virtualenv/blob/master/src/virtualenv/create/via_global_ref/builtin/python2/site.py#L90

[gaborbernat]

@jd can you make a PR addressing the issue?

[jd]

@gaborbernat sure, let me do that

[jd]

Two things:

1. the current code uses startswith which can trigger a different set of problem:

➔ export PYTHONPATH=$PWD/venv2006helloworld
➔ venv2006/bin/python -c 'import sys, pprint;pprint.pprint(sys.path)'
['',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7helloworld',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/usr/local/Cellar/python@2/2.7.17_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/Users/jd/venv2006/lib/python2.7/site-packages']

This is clearly wrong. startswith cannot be used with paths, it's too naive.

2. I can't write code that bypass PYTHONPATH because it'd need to access os.environ and os cannot be loaded until the sys.path rewrites happen.

Hint?

[gaborbernat]

For 1 we just need to validate the path ends or not with the correct. We don't have os path so string manipulation is all we can do. For 2 you need a two phase approach. First let the rewrite happen. Now import is and fixup pythonpath entries.

[jd]

Gave a try at this in #1673

[gaborbernat]

I like the idea, would just encourage some style fixes 👍

[nsoranzo]

I've encountered this too and can be workarounded by pinning virtualenv to 20.0.4, so it was likely introduced in #1641 .

[gaborbernat]

Well, it was working before #1641 only by chance of a technicality of how macOs works, so in my eyes is a regression of the 20.0.0 series, that being said we'll have out a fix next weeks second part.

[gaborbernat]

Resolved by #1673