terryfy macOS 2.7.17 and virtualenv 20.0: unable to find 'coverage' command (zope.interface)

[jamadden]

(We found a workaround so this isn't super critical, but several other projects also would have to change to use the workaround.)

zope.interface uses https://github.com/MacPython/terryfy to install the python.org distribution of Python on Travis macOS for testing for 2.7, and 3.5+. Here's the relevant .travis.yml configuration for the builds discussed. terrypy

A few days ago, there was a successful build of all versions using virtualenv 16.7.9. (Build log).

This morning, a round of builds installed virtualenv 20.0. While all the Python 3 builds worked, Python 2.7.17 failed. It seems that the $PATH wasn't setup correctly, and so the coverage command provided by the coverage wheel wasn't available. This build failed repeatedly, so it doesn't seem to be a simple fluke. (Build log). As far as I can see, that's the only relevant difference.

terryfy uses the get_macpython_environment function from https://github.com/matthew-brett/multibuild to create and activate the virtual environment after installing Python:

function get_macpython_environment {
    # Set up MacPython environment
    # Parameters:
    #     $version : [implementation-]major[.minor[.patch]]
    #         The Python implementation to install, e.g. "3.6" or "pypy-5.4"
    #     $venv_dir : {directory_name|not defined}
    #         If defined - make virtualenv in this directory, set python / pip
    #         commands accordingly
    #     $py_osx_ver: {major.minor | not defined}
    #         if defined, the macOS version that Python is built for, e.g.
    #         "10.6" or "10.9", if not defined, use the version from MB_PYTHON_OSX_VER
    #
    # Installs Python
    # Sets $PYTHON_EXE to path to Python executable
    # Sets $PIP_CMD to full command for pip (including sudo if necessary)
    # If $venv_dir defined, Sets $VIRTUALENV_CMD to virtualenv executable
    # Puts directory of $PYTHON_EXE on $PATH
    local version=$1
    local venv_dir=$2
    local py_osx_ver=${3:-$MB_PYTHON_OSX_VER}

    if [ "$USE_CCACHE" == "1" ]; then
        activate_ccache
    fi

    remove_travis_ve_pip
    install_macpython $version $py_osx_ver
    install_pip

    if [ -n "$venv_dir" ]; then
        install_virtualenv
        make_workon_venv $venv_dir
        source $venv_dir/bin/activate
    else
        export PATH="`dirname $PYTHON_EXE`:$PATH"
    fi
    export PYTHON_EXE PIP_CMD

We're calling terryfy and passing a venv_dir, and I verified that in fact the virtualenv Python is the Python that's active.

before_install:
  - |
    if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      git clone https://github.com/MacPython/terryfy
      source terryfy/travis_tools.sh
      get_python_environment $TERRYFY_PYTHON venv
    fi

With virtualenv 20.0, we get lots of errors from pip install coverage about scripts not being on the PATH:

Installing collected packages: chardet, idna, pycparser, cffi, ipaddress, enum34, cryptography, pyOpenSSL, urllib3, requests, coverage, docopt, coveralls
202  WARNING: The script chardetect is installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
203  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
204  WARNING: The scripts coverage, coverage-2.7 and coverage2 are installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
205  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
206  WARNING: The script coveralls is installed in '/Library/Frameworks/Python.framework/Versions/2.7/bin' which is not on PATH.
207  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

Is this an expected change of some sort? Does terryfy/multibuild need to adapt? Are we doing something wrong? (Possibly https://github.com/pypa/virtualenv/pull/1550/files might be related?) Thanks for any insight!

Workaround

In a subsequent build, I modified the commands in .travis.yml to use python -m pip install... and python -m coverage run …. The python -m pip install command still produced warnings about scripts not being on the PATH, but the python -m coverage run … command executed as expected. That will let us limp along.

[gaborbernat]

Thanks for the insight, I'll try to reproduce and see if I can pinpoint the issue. From the description, it does not seems to be #1550.

[matthew-brett]

Hi - just another plea for a fix for this one - it's causing some stress for those of us using CI to test Python 2.7 builds - see https://github.com/matthew-brett/multibuild/issues/305

Can I help to reproduce?

[gaborbernat]

If you can reproduce with a small example that would indeed help.

[matthew-brett]

Ouch - sorry - I can't even get that far:

curl -LO https://www.python.org/ftp/python/2.7.17/python-2.7.17-macosx10.6.pkg
sudo installer -pkg python-2.7.17-macosx10.6.pkg -target /
virtualenv --python=/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 venv
. venv/bin/activate
pip --version

gives:

Traceback (most recent call last):
  File "/Users/brettmz/Downloads/venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ImportError: No module named main

[gaborbernat]

@matthew-brett for new we need to do #1561, have #1641 open for it, it's a WIP. That might fix it but not sure hence why opening this. That being said I haven't seen that error before 🤔

As someone who removed /Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 any way to install it back?

[matthew-brett]

Does the curl, installer sequence above not reinstall that Python binary?

[gaborbernat]

It does thanks 👍

[nsoranzo]

@matthew-brett This looks similar to #1577 and what I saw in https://travis-ci.org/nsoranzo/galaxy/builds/649733024

[gaborbernat]

so looked into this; seems MacOs customizes framework builds so much that they just don't work without rewriting the mach-o (the magic from here https://github.com/pypa/virtualenv/blob/legacy/virtualenv.py#L2492-L2631); will take a while to get this done, but I'm on it 🤷‍♂

[nsoranzo]

@gaborbernat Thanks, let me know if you want me to test anything!

[matthew-brett]

Thanks too - likewise - let me know if I can help.

[gaborbernat]

Thanks, will let you know; maybe tomorrow at around now I'll have something ready.

[gaborbernat]

If anyone can give #1641 a try that would be great; seems to work for me locally but I might be missing something 🤷‍♂

[matthew-brett]

Works for me - thanks for the quick fix.

[gaborbernat]

The CI seems to still cough; I'll try to fix that to and then hopefully do a release late today.

[matthew-brett]

Great - thanks again.

[gaborbernat]

Should be fixed with https://github.com/pypa/virtualenv/pull/1641/files.

[matthew-brett]

Thanks - and thanks for the quick release to PyPI.