From 5a6ac930563bd14d558481698609764961953d36 Mon Sep 17 00:00:00 2001 From: Eric Arellano Date: Thu, 31 Jan 2019 10:01:52 -0700 Subject: [PATCH] Add ability to specify Python minor version when running `./pants` (#7187) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Problem There are three issues when trying to run `./pants3` and switching between Python 3.6 and Python 3.7. 1) You must first `rm -rf build-support/pants_dev_deps.py3.venv` and then `rm -rf ~/.cache/pants/python_cache/interpreters` because the cache will have a bad symlink to the prior Python 3 interpreter. This is inconvenient, and the latter is surprising and difficult to remember. 2) If both Python 3.6 and Python 3.7 are discoverable on the `PATH`, there is no reliable way to specify that you want to use Python 3.7. The `virtualenv` code works by invoking `python3`, which defaults in most systems to the minimum version. The only workaround is to ensure `python3` points first to `python3.7` by messing with the PATH. While this is possible, it is not obvious to the user and it would be best to provide a solution that does not require messing with the PATH. 3) When running `./pants3`, we always override `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` to allow Python 3.6, so it is impossible at the moment to set a tighter constraint without modifying source code. Even if we resolve issues 1 and 2, we must address this issue to ensure that spawned subprocesses use the correct interpreter. ## Solution ### Part 1 - renamed `venv` folders Rename `py2.venv` -> `py27.venv` and split out `py3.venv` into `py36.venv` and `py37.venv`. This allows us to have a distinct venv for each Python version, which is necessary to avoid redownloading/resymlinking dependencies anytime we change from 3.6 to 3.7. It is also necessary to fix problem #1 of having to run `rm -rf build-support/pants_dev_deps.py3.venv ~/.cache/pants/python_cache/interpreters` whenever making this change. ### Part 2 - `$PY` env var We refactor `virtualenv` and `pants_venv` to require the caller to pre-set the env var `$PY`. This value stores the bin name, such as `python3.7`, to be used in setting up the venv. If the user wants to constrain `./pants3` to a specific Python 3 interpreter, they may do so by prefixing the command with `PY=python3.7 ./pants3`. Otherwise, `./pants` will default to `python2.7` and `./pants3` will default to `python3` as before. ### Also changed - renames in `.travis.yml` Update `ci.sh` and `.travis.yml` to specify the exact Python versions as Python 2.7 and 3.6, rather than Python 2 and Python 3. `ci.sh` does make a semantic change in that it now requires exactly Python 3.6 for Py3 shards, whereas earlier it only used Py3.6 de facto. Meanwhile, `.travis.yml` changes the OSX rust tests to use Python 2.7 to fix an issue with requiring exactly Python 3.6, updates the cache directories to use the new `venv` naming scheme, and renames `py2`->`py27` and `py3`->`py36` everywhere. This is pre-work to make the Python 3.7 CI pull request easier to review. ## Alternative solutions considered ### `./pants3` interpreter flags Originally this PR added the CI flags `./pants -6` and `./pants -7` to allow forcing Pants to use only Python 3.6 or 3.7, respectively. This solution was rejected for being over-constrained and too complex. ### `./pants37` Originally we considered adding a new script `./pants37` to specify Python 3.7, rather than `./pants3 -7`. This was rejected because it results in too many root-level scripts that are prefixed `./pants`, which is confusing. Further, it is not often that Pants devs will need to switch between Python 3.6 and 3.7—we certainly will not expect them to do this frequently (whereas we do encourage frequently changing between `./pants` and `./pants3`). We merely want the ability to run with Python 3.7, particularly for CI; it is not important for the option to be especially discoverable. ## Result Calling `./pants` or `./pants3` for the first time will delete the legacy venv folders and use the new naming scheme. Scripts now behave as follows: * `./pants` will use Python 2.7 as before * `./pants3` will use the first `python3` bin on the PATH, as before * `PY=python3.6 ./pants3` will use only Python 3.6 * `PY=python3.7 ./pants3` will use only Python 3.7 Switching between these options will trigger an engine rebuild the first time, and then after that will change the interpreter used with no cost. --- .travis.yml | 506 ++++++++++++----------- build-support/bin/ci.sh | 20 +- build-support/pants_venv | 33 +- build-support/travis/travis.yml.mustache | 284 ++++++------- build-support/virtualenv | 12 +- pants | 3 + pants3 | 16 +- src/rust/engine/src/cffi_build.rs | 2 +- 8 files changed, 453 insertions(+), 423 deletions(-) diff --git a/.travis.yml b/.travis.yml index 5579f7b1032..707a37ef2df 100644 --- a/.travis.yml +++ b/.travis.yml @@ -54,8 +54,9 @@ native_engine_cache_config: &native_engine_cache_config timeout: 500 directories: - ${HOME}/.cache/pants/rust/cargo - - build-support/pants_dev_deps.py2.venv - - build-support/pants_dev_deps.py3.venv + - build-support/pants_dev_deps.py27.venv + - build-support/pants_dev_deps.py36.venv + - build-support/pants_dev_deps.py37.venv - src/rust/engine/target # Travis cache config for jobs that run a bootstrapped pants.pex. @@ -116,11 +117,11 @@ base_linux_config: &base_linux_config after_failure: - ./build-support/bin/ci-failure.sh -py2_linux_config: &py2_linux_config +py27_linux_config: &py27_linux_config <<: *base_linux_config python: &python2_version "2.7" -py3_linux_config: &py3_linux_config +py36_linux_config: &py36_linux_config <<: *base_linux_config python: &python3_version "3.6" @@ -137,19 +138,19 @@ base_linux_test_config: &base_linux_test_config before_script: - ./build-support/bin/get_ci_bootstrapped_pants_pex.sh ${BOOTSTRAPPED_PEX_BUCKET} ${BOOTSTRAPPED_PEX_KEY_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_linux_test_config: &py2_linux_test_config - <<: *py2_linux_config +py27_linux_test_config: &py27_linux_test_config + <<: *py27_linux_config <<: *base_linux_test_config stage: *cron env: - - &py2_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.linux + - &py27_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux -py3_linux_test_config: &py3_linux_test_config - <<: *py3_linux_config +py36_linux_test_config: &py36_linux_test_config + <<: *py36_linux_config <<: *base_linux_test_config stage: *test env: - - &py3_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.linux + - &py36_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux base_osx_config: &base_osx_config os: osx @@ -159,18 +160,18 @@ base_osx_config: &base_osx_config - chmod 755 /usr/local/bin/jq - ./build-support/bin/install_aws_cli_for_ci.sh -py2_osx_config: &py2_osx_config +py27_osx_config: &py27_osx_config <<: *base_osx_config -py3_osx_config: &py3_osx_config +py36_osx_config: &py36_osx_config <<: *base_osx_config addons: brew: - packages: &py3_osx_config_brew_packages + packages: &py36_osx_config_brew_packages - openssl env: # Fix Python 3 issue linking to OpenSSL - - &py3_osx_config_env > + - &py36_osx_config_env > PATH="/usr/local/opt/openssl/bin:$PATH" LDFLAGS="-L/usr/local/opt/openssl/lib" CPPFLAGS="-I/usr/local/opt/openssl/include" @@ -193,26 +194,26 @@ base_osx_test_config: &base_osx_test_config - ulimit -n 8192 - ./build-support/bin/get_ci_bootstrapped_pants_pex.sh ${BOOTSTRAPPED_PEX_BUCKET} ${BOOTSTRAPPED_PEX_KEY_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_osx_test_config: &py2_osx_test_config - <<: *py2_osx_config +py27_osx_test_config: &py27_osx_test_config + <<: *py27_osx_config <<: *base_osx_test_config stage: *cron env: - - &py2_osx_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.osx + - &py27_osx_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.osx -py3_osx_test_config: &py3_osx_test_config - <<: *py3_osx_config +py36_osx_test_config: &py36_osx_test_config + <<: *py36_osx_config <<: *base_osx_test_config stage: *test env: - # Must duplicate py3_osx_config's env because it cannot be merged into a new anchor - - &py3_osx_test_config_env > + # Must duplicate py36_osx_config's env because it cannot be merged into a new anchor + - &py36_osx_test_config_env > PATH="/usr/local/opt/openssl/bin:$PATH" LDFLAGS="-L/usr/local/opt/openssl/lib" CPPFLAGS="-I/usr/local/opt/openssl/include" PYENV_ROOT="${HOME}/.pyenv" PATH="${PYENV_ROOT}/shims:${PATH}" - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.osx + BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.osx linux_with_fuse: &linux_with_fuse before_install: @@ -262,25 +263,25 @@ base_linux_build_engine: &base_linux_build_engine sh -c "./build-support/bin/ci.sh ${BOOTSTRAP_ARGS} && ./build-support/bin/release.sh -f" - aws --no-sign-request --region us-east-1 s3 cp ${TRAVIS_BUILD_DIR}/pants.pex ${BOOTSTRAPPED_PEX_URL_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_linux_build_engine: &py2_linux_build_engine - <<: *py2_linux_config +py27_linux_build_engine: &py27_linux_build_engine + <<: *py27_linux_config <<: *base_linux_build_engine - name: "Build Linux native engine and pants.pex (Py2 PEX)" + name: "Build Linux native engine and pants.pex (Py2.7 PEX)" env: - # NB: Only the Py2 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: + # NB: Only the Py2.7 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: # either linux shard could upload this binary, since it does not depend on python at all. - PREPARE_DEPLOY=1 - - CACHE_NAME=linuxpexbuild.py2 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.linux + - CACHE_NAME=linuxpexbuild.py27 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux - BOOTSTRAP_ARGS='-2b' -py3_linux_build_engine: &py3_linux_build_engine - <<: *py3_linux_config +py36_linux_build_engine: &py36_linux_build_engine + <<: *py36_linux_config <<: *base_linux_build_engine - name: "Build Linux native engine and pants.pex (Py3 PEX)" + name: "Build Linux native engine and pants.pex (Py3.6 PEX)" env: - - CACHE_NAME=linuxpexbuild.py3 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.linux + - CACHE_NAME=linuxpexbuild.py36 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux - BOOTSTRAP_ARGS='-b' base_osx_build_engine: &base_osx_build_engine @@ -299,46 +300,46 @@ base_osx_build_engine: &base_osx_build_engine after_failure: - ./build-support/bin/ci-failure.sh -py2_osx_build_engine: &py2_osx_build_engine - <<: *py2_osx_config +py27_osx_build_engine: &py27_osx_build_engine + <<: *py27_osx_config <<: *base_osx_build_engine - name: "Build OSX native engine and pants.pex (Py2 PEX)" + name: "Build OSX native engine and pants.pex (Py2.7 PEX)" env: - *base_osx_build_engine_env - - CACHE_NAME=osxpexbuild.py2 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.osx + - CACHE_NAME=osxpexbuild.py27 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.osx - BOOTSTRAP_ARGS='-2b' -py3_osx_build_engine: &py3_osx_build_engine - <<: *py3_osx_config +py36_osx_build_engine: &py36_osx_build_engine + <<: *py36_osx_config <<: *base_osx_build_engine - name: "Build OSX native engine and pants.pex (Py3 PEX)" + name: "Build OSX native engine and pants.pex (Py3.6 PEX)" env: - *base_osx_build_engine_env - - *py3_osx_config_env - - CACHE_NAME=osxpexbuild.py3 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.osx + - *py36_osx_config_env + - CACHE_NAME=osxpexbuild.py36 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.osx - BOOTSTRAP_ARGS='-b' # ------------------------------------------------------------------------- # Lint # ------------------------------------------------------------------------- -py2_lint: &py2_lint - <<: *py2_linux_test_config - name: "Self-checks and lint (Py2 PEX)" +py27_lint: &py27_lint + <<: *py27_linux_test_config + name: "Self-checks and lint (Py2.7 PEX)" env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxselfchecks.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxselfchecks.py27 script: - ./build-support/bin/travis-ci.sh -fmrt2 -py3_lint: &py3_lint - <<: *py3_linux_test_config - name: "Self-checks and lint (Py3 PEX)" +py36_lint: &py36_lint + <<: *py36_linux_test_config + name: "Self-checks and lint (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxselfchecks.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxselfchecks.py36 script: - ./build-support/bin/travis-ci.sh -fmrt @@ -387,13 +388,13 @@ base_build_wheels: &base_build_wheels linux_build_wheels: &linux_build_wheels # Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize - # compatibility. This is a Py2 shard, so it is not subject to #6985. + # compatibility. This is a Py2.7 shard, so it is not subject to #6985. <<: *travis_docker_image - <<: *py2_linux_test_config + <<: *py27_linux_test_config <<: *base_build_wheels name: "Build Linux wheels (No PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - *base_build_wheels_env - CACHE_NAME=linuxwheelsbuild script: @@ -405,12 +406,12 @@ linux_build_wheels: &linux_build_wheels sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" osx_build_wheels: &osx_build_wheels - <<: *py2_osx_test_config + <<: *py27_osx_test_config <<: *base_build_wheels name: "Build OSX wheels (No PEX)" osx_image: xcode8 env: - - *py2_osx_test_config_env + - *py27_osx_test_config_env - *base_build_wheels_env - CACHE_NAME=osxwheelsbuild script: @@ -454,7 +455,10 @@ osx_rust_tests: &osx_rust_tests casks: - osxfuse script: - - ./build-support/bin/travis-ci.sh -e + # N.B. We run this with Python 2 because this osx_image does not have + # Python 3.6 in its environment. We do not care which Python version + # we use and do not want to incur the cost of using pyenv to get 3.6. + - ./build-support/bin/travis-ci.sh -e2 # ------------------------------------------------------------------------- # OSX sanity checks @@ -469,61 +473,61 @@ base_osx_10_12_sanity_check: &base_osx_10_12_sanity_check <<: *base_osx_sanity_check osx_image: xcode9.2 -py2_osx_10_12_sanity_check: &py2_osx_10_12_sanity_check - <<: *py2_osx_test_config +py27_osx_10_12_sanity_check: &py27_osx_10_12_sanity_check + <<: *py27_osx_test_config <<: *base_osx_10_12_sanity_check - name: "OSX 10.12 sanity check (Py2 PEX)" + name: "OSX 10.12 sanity check (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macos10.12sanity.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macos10.12sanity.py27 -py3_osx_10_12_sanity_check: &py3_osx_10_12_sanity_check - <<: *py3_osx_test_config +py36_osx_10_12_sanity_check: &py36_osx_10_12_sanity_check + <<: *py36_osx_test_config <<: *base_osx_10_12_sanity_check - name: "OSX 10.12 sanity check (Py3 PEX)" + name: "OSX 10.12 sanity check (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macos10.12sanity.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macos10.12sanity.py36 base_osx_10_13_sanity_check: &base_osx_10_13_sanity_check <<: *base_osx_sanity_check osx_image: xcode10.1 -py2_osx_10_13_sanity_check: &py2_osx_10_13_sanity_check - <<: *py2_osx_test_config +py27_osx_10_13_sanity_check: &py27_osx_10_13_sanity_check + <<: *py27_osx_test_config <<: *base_osx_10_13_sanity_check - name: "OSX 10.13 sanity check (Py2 PEX)" + name: "OSX 10.13 sanity check (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macos10.13sanity.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macos10.13sanity.py27 -py3_osx_10_13_sanity_check: &py3_osx_10_13_sanity_check - <<: *py3_osx_test_config +py36_osx_10_13_sanity_check: &py36_osx_10_13_sanity_check + <<: *py36_osx_test_config <<: *base_osx_10_13_sanity_check - name: "OSX 10.13 sanity check (Py3 PEX)" + name: "OSX 10.13 sanity check (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macos10.13sanity.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macos10.13sanity.py36 # ------------------------------------------------------------------------- # Platform specific tests # ------------------------------------------------------------------------- -py2_osx_platform_tests: &py2_osx_platform_tests - <<: *py2_osx_test_config - name: "OSX platform-specific tests (Py2 PEX)" +py27_osx_platform_tests: &py27_osx_platform_tests + <<: *py27_osx_test_config + name: "OSX platform-specific tests (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macosplatformtests.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macosplatformtests.py27 script: - ./build-support/bin/travis-ci.sh -z2 -py3_osx_platform_tests: &py3_osx_platform_tests - <<: *py3_osx_test_config - name: "OSX platform-specific tests (Py3 PEX)" +py36_osx_platform_tests: &py36_osx_platform_tests + <<: *py36_osx_test_config + name: "OSX platform-specific tests (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macosplatformtests.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macosplatformtests.py36 script: - ./build-support/bin/travis-ci.sh -z @@ -534,23 +538,23 @@ py3_osx_platform_tests: &py3_osx_platform_tests base_jvm_tests: &base_jvm_tests <<: *linux_with_fuse -py2_jvm_tests: &py2_jvm_tests - <<: *py2_linux_test_config +py27_jvm_tests: &py27_jvm_tests + <<: *py27_linux_test_config <<: *base_jvm_tests - name: "JVM tests (Py2 PEX)" + name: "JVM tests (Py2.7 PEX)" env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxjvmtests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxjvmtests.py27 script: - ./build-support/bin/travis-ci.sh -j2 -py3_jvm_tests: &py3_jvm_tests - <<: *py3_linux_test_config +py36_jvm_tests: &py36_jvm_tests + <<: *py36_linux_test_config <<: *base_jvm_tests - name: "JVM tests (Py3 PEX)" + name: "JVM tests (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxjvmtests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxjvmtests.py36 script: - ./build-support/bin/travis-ci.sh -j @@ -601,14 +605,14 @@ deploy_unstable_multiplatform_pex: &deploy_unstable_multiplatform_pex matrix: include: - - <<: *py2_linux_build_engine - - <<: *py3_linux_build_engine + - <<: *py27_linux_build_engine + - <<: *py36_linux_build_engine - - <<: *py2_osx_build_engine - - <<: *py3_osx_build_engine + - <<: *py27_osx_build_engine + - <<: *py36_osx_build_engine - - <<: *py2_lint - - <<: *py3_lint + - <<: *py27_lint + - <<: *py36_lint - <<: *linux_rust_clippy - <<: *cargo_audit @@ -616,341 +620,341 @@ matrix: - <<: *linux_build_wheels - <<: *osx_build_wheels - - <<: *py2_linux_test_config - name: "Unit tests for pants and pants-plugins (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Unit tests for pants and pants-plugins (Py2.7 PEX)" stage: *test env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxunittests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxunittests.py27 script: - ./build-support/bin/travis-ci.sh -2lp - - <<: *py3_linux_test_config - name: "Unit tests for pants and pants-plugins (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Unit tests for pants and pants-plugins (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxunittests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxunittests.py36 script: - ./build-support/bin/travis-ci.sh -lp - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 0 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 0 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard0 script: - ./build-support/bin/travis-ci.sh -c -i 0/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 1 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 1 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard1 script: - ./build-support/bin/travis-ci.sh -c -i 1/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 2 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 2 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard2 script: - ./build-support/bin/travis-ci.sh -c -i 2/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 3 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 3 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard3 script: - ./build-support/bin/travis-ci.sh -c -i 3/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 4 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 4 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard4 script: - ./build-support/bin/travis-ci.sh -c -i 4/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 5 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 5 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard5 script: - ./build-support/bin/travis-ci.sh -c -i 5/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 6 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 6 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard6 script: - ./build-support/bin/travis-ci.sh -c -i 6/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 7 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 7 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard7 script: - ./build-support/bin/travis-ci.sh -c -i 7/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 8 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 8 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard8 script: - ./build-support/bin/travis-ci.sh -c -i 8/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 9 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 9 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard9 script: - ./build-support/bin/travis-ci.sh -c -i 9/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 10 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 10 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard10 script: - ./build-support/bin/travis-ci.sh -c -i 10/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 11 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 11 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard11 script: - ./build-support/bin/travis-ci.sh -c -i 11/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 12 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 12 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard12 script: - ./build-support/bin/travis-ci.sh -c -i 12/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 13 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 13 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard13 script: - ./build-support/bin/travis-ci.sh -c -i 13/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 14 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 14 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard14 script: - ./build-support/bin/travis-ci.sh -c -i 14/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 15 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 15 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard15 script: - ./build-support/bin/travis-ci.sh -c -i 15/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 16 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 16 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard16 script: - ./build-support/bin/travis-ci.sh -c -i 16/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 17 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 17 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard17 script: - ./build-support/bin/travis-ci.sh -c -i 17/19 - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard 18 (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard 18 (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard18 script: - ./build-support/bin/travis-ci.sh -c -i 18/19 - - <<: *py2_linux_test_config - name: "Blacklisted integration tests for pants - shard 0 (Py2 PEX w/ Py3 constraints)" + - <<: *py27_linux_test_config + name: "Blacklisted integration tests for pants - shard 0 (Py2.7 PEX w/ Py3.6 constraints)" stage: *test env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="['CPython>=3.6']" - - CACHE_NAME=integrationshard0.py2blacklist + - CACHE_NAME=integrationshard0.py27blacklist script: - ./build-support/bin/travis-ci.sh -c2w -i 0/1 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 0 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 0 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard0 script: - ./build-support/bin/travis-ci.sh -c2 -i 0/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 1 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 1 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard1 script: - ./build-support/bin/travis-ci.sh -c2 -i 1/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 2 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 2 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard2 script: - ./build-support/bin/travis-ci.sh -c2 -i 2/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 3 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 3 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard3 script: - ./build-support/bin/travis-ci.sh -c2 -i 3/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 4 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 4 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard4 script: - ./build-support/bin/travis-ci.sh -c2 -i 4/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 5 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 5 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard5 script: - ./build-support/bin/travis-ci.sh -c2 -i 5/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 6 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 6 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard6 script: - ./build-support/bin/travis-ci.sh -c2 -i 6/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 7 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 7 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard7 script: - ./build-support/bin/travis-ci.sh -c2 -i 7/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 8 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 8 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard8 script: - ./build-support/bin/travis-ci.sh -c2 -i 8/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 9 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 9 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard9 script: - ./build-support/bin/travis-ci.sh -c2 -i 9/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 10 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 10 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard10 script: - ./build-support/bin/travis-ci.sh -c2 -i 10/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 11 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 11 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard11 script: - ./build-support/bin/travis-ci.sh -c2 -i 11/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 12 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 12 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard12 script: - ./build-support/bin/travis-ci.sh -c2 -i 12/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 13 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 13 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard13 script: - ./build-support/bin/travis-ci.sh -c2 -i 13/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 14 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 14 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard14 script: - ./build-support/bin/travis-ci.sh -c2 -i 14/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 15 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 15 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard15 script: - ./build-support/bin/travis-ci.sh -c2 -i 15/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 16 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 16 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard16 script: - ./build-support/bin/travis-ci.sh -c2 -i 16/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 17 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 17 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard17 script: - ./build-support/bin/travis-ci.sh -c2 -i 17/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 18 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 18 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard18 script: - ./build-support/bin/travis-ci.sh -c2 -i 18/20 - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard 19 (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard 19 (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard19 script: - ./build-support/bin/travis-ci.sh -c2 -i 19/20 @@ -959,34 +963,34 @@ matrix: - <<: *linux_rust_tests - <<: *osx_rust_tests - - <<: *py2_linux_test_config - name: "Python contrib tests (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Python contrib tests (Py2.7 PEX)" stage: *test env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxcontribtests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxcontribtests.py27 script: - ./build-support/bin/travis-ci.sh -2n - - <<: *py3_linux_test_config - name: "Python contrib tests (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Python contrib tests (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxcontribtests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxcontribtests.py36 script: - ./build-support/bin/travis-ci.sh -n - - <<: *py2_osx_10_12_sanity_check - - <<: *py3_osx_10_12_sanity_check + - <<: *py27_osx_10_12_sanity_check + - <<: *py36_osx_10_12_sanity_check - - <<: *py2_osx_10_13_sanity_check - - <<: *py3_osx_10_13_sanity_check + - <<: *py27_osx_10_13_sanity_check + - <<: *py36_osx_10_13_sanity_check - - <<: *py2_osx_platform_tests - - <<: *py3_osx_platform_tests + - <<: *py27_osx_platform_tests + - <<: *py36_osx_platform_tests - - <<: *py2_jvm_tests - - <<: *py3_jvm_tests + - <<: *py27_jvm_tests + - <<: *py36_jvm_tests - <<: *deploy_stable_multiplatform_pex - <<: *deploy_unstable_multiplatform_pex diff --git a/build-support/bin/ci.sh b/build-support/bin/ci.sh index b350a141282..7f294494f65 100755 --- a/build-support/bin/ci.sh +++ b/build-support/bin/ci.sh @@ -15,7 +15,7 @@ Runs commons tests for local or hosted CI. Usage: $0 (-h|-2fxbkmrjlpuneycitzsw) -h print out this help message - -2 Run using Python 2 (defaults to using Python 3). + -2 Run using Python 2.7 (defaults to using Python 3.6). -f run python code formatting checks -x run bootstrap clean-all (assume bootstrapping from a fresh clone) @@ -58,7 +58,6 @@ EOF python_unit_shard="0/1" python_contrib_shard="0/1" python_intg_shard="0/1" -python_two="false" while getopts "h2fxbmrjlpeasu:ny:ci:tzw" opt; do case ${opt} in @@ -105,17 +104,22 @@ esac # We're running against a Pants clone. export PANTS_DEV=1 -# Determine which Python interpreter to use for bootstrapping pex and for executing subprocesses. -# Order matters here. We must constrain subprocesses before running the bootstrap stage, -# or we will encounter the _Py_Dealloc error when bootstrapping a Python 3 PEX. +# Note that we set PY, and when running with Python 3, also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS. +# This would usually not be necessary when developing locally, because the `./pants` and `./pants3` +# scripts set these constraints for us already. However, we must set the values here because in non-bootstrap shards +# we run CI using `./pants.pex` instead of the scripts `./pants` and `./pants3`, so those scripts cannot set +# the relevant environment variables. Without setting these environment variables, the Python 3 shards will try to +# execute subprocesses using Python 2, which results in the _Py_Dealloc error (#6985), and shards that do not +# pull down `./pants.pex` but still use a virtualenv (such as Rust Tests) will fail to execute. if [[ "${python_two:-false}" == "false" ]]; then - py_version_number="3" + py_version_number="3.6" bootstrap_pants_script="./pants3" - export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS='["CPython>=3.6,<4"]' + export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="['CPython==${py_version_number}.*']" else - py_version_number="2" + py_version_number="2.7" bootstrap_pants_script="./pants" fi +export PY="python${py_version_number}" banner "Using Python ${py_version_number} to execute spawned subprocesses (e.g. tests)" if [[ "${run_bootstrap:-false}" == "true" ]]; then diff --git a/build-support/pants_venv b/build-support/pants_venv index 47a344549e9..457d4809830 100755 --- a/build-support/pants_venv +++ b/build-support/pants_venv @@ -11,26 +11,35 @@ REQUIREMENTS=( ${REPO_ROOT}/pants-plugins/3rdparty/python/requirements.txt ) +venv_dir_prefix="${REPO_ROOT}/build-support/pants_dev_deps" + function venv_dir() { - py_version="py2"; [ "${PANTS_USE_PYTHON3}" = true ] && py_version="py3" - echo ${REPO_ROOT}/build-support/pants_dev_deps.${py_version}.venv + py_venv_version=$(${PY} -c 'import sys; print("".join(map(str, sys.version_info[0:2])))') + echo "${venv_dir_prefix}.py${py_venv_version}.venv" } function activate_venv() { source "$(venv_dir)/bin/activate" } -function create_venv() { +function remove_venv() { rm -rf "$(venv_dir)" - # If `pants_dev_deps.venv` still exists, delete both its legacy folder and the cached - # Python interpreter folder, which contains problematic symlinks. - # Note we only perform these removals for the first time, to avoid continually deleting - # the cache when switching between using `pants_dev_deps.py2.venv` and `pants_dev_deps.py3.venv`. - legacy_venv_dir="${REPO_ROOT}/build-support/pants_dev_deps.venv" - if [ -d "${legacy_venv_dir}" ]; then - rm -rf "${legacy_venv_dir}" - rm -rf "${HOME}/.cache/pants/python_cache/interpreters" - fi + # If `pants_dev_deps.venv` or `pants_dev_deps.py{2,3}.venv` still exist, delete both the legacy folders + # and the cached Python interpreter folder, which contains problematic symlinks. Note we only perform + # these removals for the first time, to avoid continually deleting the cache when switching + # between valid venvs. + legacy_venv_dirs=("${venv_dir_prefix}.venv" "${venv_dir_prefix}.py2.venv" "${venv_dir_prefix}.py3.venv") + for legacy_venv_dir in "${legacy_venv_dirs[@]}"; do + if [ -d "${legacy_venv_dir}" ]; then + rm -rf "${legacy_venv_dirs[@]}" + rm -rf "${HOME}/.cache/pants/python_cache/interpreters" + break + fi + done +} + +function create_venv() { + remove_venv "${REPO_ROOT}/build-support/virtualenv" "$(venv_dir)" } diff --git a/build-support/travis/travis.yml.mustache b/build-support/travis/travis.yml.mustache index 0dab1f3d310..08d48816df3 100644 --- a/build-support/travis/travis.yml.mustache +++ b/build-support/travis/travis.yml.mustache @@ -47,8 +47,9 @@ native_engine_cache_config: &native_engine_cache_config timeout: 500 directories: - ${HOME}/.cache/pants/rust/cargo - - build-support/pants_dev_deps.py2.venv - - build-support/pants_dev_deps.py3.venv + - build-support/pants_dev_deps.py27.venv + - build-support/pants_dev_deps.py36.venv + - build-support/pants_dev_deps.py37.venv - src/rust/engine/target # Travis cache config for jobs that run a bootstrapped pants.pex. @@ -109,11 +110,11 @@ base_linux_config: &base_linux_config after_failure: - ./build-support/bin/ci-failure.sh -py2_linux_config: &py2_linux_config +py27_linux_config: &py27_linux_config <<: *base_linux_config python: &python2_version "2.7" -py3_linux_config: &py3_linux_config +py36_linux_config: &py36_linux_config <<: *base_linux_config python: &python3_version "3.6" @@ -125,19 +126,19 @@ base_linux_test_config: &base_linux_test_config before_script: - ./build-support/bin/get_ci_bootstrapped_pants_pex.sh ${BOOTSTRAPPED_PEX_BUCKET} ${BOOTSTRAPPED_PEX_KEY_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_linux_test_config: &py2_linux_test_config - <<: *py2_linux_config +py27_linux_test_config: &py27_linux_test_config + <<: *py27_linux_config <<: *base_linux_test_config stage: *cron env: - - &py2_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.linux + - &py27_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux -py3_linux_test_config: &py3_linux_test_config - <<: *py3_linux_config +py36_linux_test_config: &py36_linux_test_config + <<: *py36_linux_config <<: *base_linux_test_config stage: *test env: - - &py3_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.linux + - &py36_linux_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux base_osx_config: &base_osx_config os: osx @@ -145,18 +146,18 @@ base_osx_config: &base_osx_config before_install: {{>before_install_osx}} -py2_osx_config: &py2_osx_config +py27_osx_config: &py27_osx_config <<: *base_osx_config -py3_osx_config: &py3_osx_config +py36_osx_config: &py36_osx_config <<: *base_osx_config addons: brew: - packages: &py3_osx_config_brew_packages + packages: &py36_osx_config_brew_packages - openssl env: # Fix Python 3 issue linking to OpenSSL - - &py3_osx_config_env > + - &py36_osx_config_env > PATH="/usr/local/opt/openssl/bin:$PATH" LDFLAGS="-L/usr/local/opt/openssl/lib" CPPFLAGS="-I/usr/local/opt/openssl/include" @@ -177,26 +178,26 @@ base_osx_test_config: &base_osx_test_config - ulimit -n 8192 - ./build-support/bin/get_ci_bootstrapped_pants_pex.sh ${BOOTSTRAPPED_PEX_BUCKET} ${BOOTSTRAPPED_PEX_KEY_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_osx_test_config: &py2_osx_test_config - <<: *py2_osx_config +py27_osx_test_config: &py27_osx_test_config + <<: *py27_osx_config <<: *base_osx_test_config stage: *cron env: - - &py2_osx_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.osx + - &py27_osx_test_config_env BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.osx -py3_osx_test_config: &py3_osx_test_config - <<: *py3_osx_config +py36_osx_test_config: &py36_osx_test_config + <<: *py36_osx_config <<: *base_osx_test_config stage: *test env: - # Must duplicate py3_osx_config's env because it cannot be merged into a new anchor - - &py3_osx_test_config_env > + # Must duplicate py36_osx_config's env because it cannot be merged into a new anchor + - &py36_osx_test_config_env > PATH="/usr/local/opt/openssl/bin:$PATH" LDFLAGS="-L/usr/local/opt/openssl/lib" CPPFLAGS="-I/usr/local/opt/openssl/include" PYENV_ROOT="${HOME}/.pyenv" PATH="${PYENV_ROOT}/shims:${PATH}" - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.osx + BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.osx linux_with_fuse: &linux_with_fuse before_install: @@ -241,25 +242,25 @@ base_linux_build_engine: &base_linux_build_engine sh -c "./build-support/bin/ci.sh ${BOOTSTRAP_ARGS} && ./build-support/bin/release.sh -f" - aws --no-sign-request --region us-east-1 s3 cp ${TRAVIS_BUILD_DIR}/pants.pex ${BOOTSTRAPPED_PEX_URL_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} -py2_linux_build_engine: &py2_linux_build_engine - <<: *py2_linux_config +py27_linux_build_engine: &py27_linux_build_engine + <<: *py27_linux_config <<: *base_linux_build_engine - name: "Build Linux native engine and pants.pex (Py2 PEX)" + name: "Build Linux native engine and pants.pex (Py2.7 PEX)" env: - # NB: Only the Py2 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: + # NB: Only the Py2.7 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: # either linux shard could upload this binary, since it does not depend on python at all. - PREPARE_DEPLOY=1 - - CACHE_NAME=linuxpexbuild.py2 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.linux + - CACHE_NAME=linuxpexbuild.py27 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.linux - BOOTSTRAP_ARGS='-2b' -py3_linux_build_engine: &py3_linux_build_engine - <<: *py3_linux_config +py36_linux_build_engine: &py36_linux_build_engine + <<: *py36_linux_config <<: *base_linux_build_engine - name: "Build Linux native engine and pants.pex (Py3 PEX)" + name: "Build Linux native engine and pants.pex (Py3.6 PEX)" env: - - CACHE_NAME=linuxpexbuild.py3 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.linux + - CACHE_NAME=linuxpexbuild.py36 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux - BOOTSTRAP_ARGS='-b' base_osx_build_engine: &base_osx_build_engine @@ -278,46 +279,46 @@ base_osx_build_engine: &base_osx_build_engine after_failure: - ./build-support/bin/ci-failure.sh -py2_osx_build_engine: &py2_osx_build_engine - <<: *py2_osx_config +py27_osx_build_engine: &py27_osx_build_engine + <<: *py27_osx_config <<: *base_osx_build_engine - name: "Build OSX native engine and pants.pex (Py2 PEX)" + name: "Build OSX native engine and pants.pex (Py2.7 PEX)" env: - *base_osx_build_engine_env - - CACHE_NAME=osxpexbuild.py2 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py2.osx + - CACHE_NAME=osxpexbuild.py27 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py27.osx - BOOTSTRAP_ARGS='-2b' -py3_osx_build_engine: &py3_osx_build_engine - <<: *py3_osx_config +py36_osx_build_engine: &py36_osx_build_engine + <<: *py36_osx_config <<: *base_osx_build_engine - name: "Build OSX native engine and pants.pex (Py3 PEX)" + name: "Build OSX native engine and pants.pex (Py3.6 PEX)" env: - *base_osx_build_engine_env - - *py3_osx_config_env - - CACHE_NAME=osxpexbuild.py3 - - BOOTSTRAPPED_PEX_KEY_SUFFIX=py3.osx + - *py36_osx_config_env + - CACHE_NAME=osxpexbuild.py36 + - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.osx - BOOTSTRAP_ARGS='-b' # ------------------------------------------------------------------------- # Lint # ------------------------------------------------------------------------- -py2_lint: &py2_lint - <<: *py2_linux_test_config - name: "Self-checks and lint (Py2 PEX)" +py27_lint: &py27_lint + <<: *py27_linux_test_config + name: "Self-checks and lint (Py2.7 PEX)" env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxselfchecks.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxselfchecks.py27 script: - ./build-support/bin/travis-ci.sh -fmrt2 -py3_lint: &py3_lint - <<: *py3_linux_test_config - name: "Self-checks and lint (Py3 PEX)" +py36_lint: &py36_lint + <<: *py36_linux_test_config + name: "Self-checks and lint (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxselfchecks.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxselfchecks.py36 script: - ./build-support/bin/travis-ci.sh -fmrt @@ -366,13 +367,13 @@ base_build_wheels: &base_build_wheels linux_build_wheels: &linux_build_wheels # Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize - # compatibility. This is a Py2 shard, so it is not subject to #6985. + # compatibility. This is a Py2.7 shard, so it is not subject to #6985. <<: *travis_docker_image - <<: *py2_linux_test_config + <<: *py27_linux_test_config <<: *base_build_wheels name: "Build Linux wheels (No PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - *base_build_wheels_env - CACHE_NAME=linuxwheelsbuild script: @@ -384,12 +385,12 @@ linux_build_wheels: &linux_build_wheels sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" osx_build_wheels: &osx_build_wheels - <<: *py2_osx_test_config + <<: *py27_osx_test_config <<: *base_build_wheels name: "Build OSX wheels (No PEX)" osx_image: xcode8 env: - - *py2_osx_test_config_env + - *py27_osx_test_config_env - *base_build_wheels_env - CACHE_NAME=osxwheelsbuild script: @@ -433,7 +434,10 @@ osx_rust_tests: &osx_rust_tests casks: - osxfuse script: - - ./build-support/bin/travis-ci.sh -e + # N.B. We run this with Python 2 because this osx_image does not have + # Python 3.6 in its environment. We do not care which Python version + # we use and do not want to incur the cost of using pyenv to get 3.6. + - ./build-support/bin/travis-ci.sh -e2 # ------------------------------------------------------------------------- # OSX sanity checks @@ -448,61 +452,61 @@ base_osx_10_12_sanity_check: &base_osx_10_12_sanity_check <<: *base_osx_sanity_check osx_image: xcode9.2 -py2_osx_10_12_sanity_check: &py2_osx_10_12_sanity_check - <<: *py2_osx_test_config +py27_osx_10_12_sanity_check: &py27_osx_10_12_sanity_check + <<: *py27_osx_test_config <<: *base_osx_10_12_sanity_check - name: "OSX 10.12 sanity check (Py2 PEX)" + name: "OSX 10.12 sanity check (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macos10.12sanity.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macos10.12sanity.py27 -py3_osx_10_12_sanity_check: &py3_osx_10_12_sanity_check - <<: *py3_osx_test_config +py36_osx_10_12_sanity_check: &py36_osx_10_12_sanity_check + <<: *py36_osx_test_config <<: *base_osx_10_12_sanity_check - name: "OSX 10.12 sanity check (Py3 PEX)" + name: "OSX 10.12 sanity check (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macos10.12sanity.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macos10.12sanity.py36 base_osx_10_13_sanity_check: &base_osx_10_13_sanity_check <<: *base_osx_sanity_check osx_image: xcode10.1 -py2_osx_10_13_sanity_check: &py2_osx_10_13_sanity_check - <<: *py2_osx_test_config +py27_osx_10_13_sanity_check: &py27_osx_10_13_sanity_check + <<: *py27_osx_test_config <<: *base_osx_10_13_sanity_check - name: "OSX 10.13 sanity check (Py2 PEX)" + name: "OSX 10.13 sanity check (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macos10.13sanity.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macos10.13sanity.py27 -py3_osx_10_13_sanity_check: &py3_osx_10_13_sanity_check - <<: *py3_osx_test_config +py36_osx_10_13_sanity_check: &py36_osx_10_13_sanity_check + <<: *py36_osx_test_config <<: *base_osx_10_13_sanity_check - name: "OSX 10.13 sanity check (Py3 PEX)" + name: "OSX 10.13 sanity check (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macos10.13sanity.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macos10.13sanity.py36 # ------------------------------------------------------------------------- # Platform specific tests # ------------------------------------------------------------------------- -py2_osx_platform_tests: &py2_osx_platform_tests - <<: *py2_osx_test_config - name: "OSX platform-specific tests (Py2 PEX)" +py27_osx_platform_tests: &py27_osx_platform_tests + <<: *py27_osx_test_config + name: "OSX platform-specific tests (Py2.7 PEX)" env: - - *py2_osx_test_config_env - - CACHE_NAME=macosplatformtests.py2 + - *py27_osx_test_config_env + - CACHE_NAME=macosplatformtests.py27 script: - ./build-support/bin/travis-ci.sh -z2 -py3_osx_platform_tests: &py3_osx_platform_tests - <<: *py3_osx_test_config - name: "OSX platform-specific tests (Py3 PEX)" +py36_osx_platform_tests: &py36_osx_platform_tests + <<: *py36_osx_test_config + name: "OSX platform-specific tests (Py3.6 PEX)" env: - - *py3_osx_test_config_env - - CACHE_NAME=macosplatformtests.py3 + - *py36_osx_test_config_env + - CACHE_NAME=macosplatformtests.py36 script: - ./build-support/bin/travis-ci.sh -z @@ -513,23 +517,23 @@ py3_osx_platform_tests: &py3_osx_platform_tests base_jvm_tests: &base_jvm_tests <<: *linux_with_fuse -py2_jvm_tests: &py2_jvm_tests - <<: *py2_linux_test_config +py27_jvm_tests: &py27_jvm_tests + <<: *py27_linux_test_config <<: *base_jvm_tests - name: "JVM tests (Py2 PEX)" + name: "JVM tests (Py2.7 PEX)" env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxjvmtests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxjvmtests.py27 script: - ./build-support/bin/travis-ci.sh -j2 -py3_jvm_tests: &py3_jvm_tests - <<: *py3_linux_test_config +py36_jvm_tests: &py36_jvm_tests + <<: *py36_linux_test_config <<: *base_jvm_tests - name: "JVM tests (Py3 PEX)" + name: "JVM tests (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxjvmtests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxjvmtests.py36 script: - ./build-support/bin/travis-ci.sh -j @@ -580,14 +584,14 @@ deploy_unstable_multiplatform_pex: &deploy_unstable_multiplatform_pex matrix: include: - - <<: *py2_linux_build_engine - - <<: *py3_linux_build_engine + - <<: *py27_linux_build_engine + - <<: *py36_linux_build_engine - - <<: *py2_osx_build_engine - - <<: *py3_osx_build_engine + - <<: *py27_osx_build_engine + - <<: *py36_osx_build_engine - - <<: *py2_lint - - <<: *py3_lint + - <<: *py27_lint + - <<: *py36_lint - <<: *linux_rust_clippy - <<: *cargo_audit @@ -595,50 +599,50 @@ matrix: - <<: *linux_build_wheels - <<: *osx_build_wheels - - <<: *py2_linux_test_config - name: "Unit tests for pants and pants-plugins (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Unit tests for pants and pants-plugins (Py2.7 PEX)" stage: *test env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxunittests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxunittests.py27 script: - ./build-support/bin/travis-ci.sh -2lp - - <<: *py3_linux_test_config - name: "Unit tests for pants and pants-plugins (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Unit tests for pants and pants-plugins (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxunittests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxunittests.py36 script: - ./build-support/bin/travis-ci.sh -lp {{#py3_integration_shards}} - - <<: *py3_linux_test_config - name: "Integration tests for pants - shard {{.}} (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Integration tests for pants - shard {{.}} (Py3.6 PEX)" env: - - *py3_linux_test_config_env + - *py36_linux_test_config_env - CACHE_NAME=integrationshard{{.}} script: - ./build-support/bin/travis-ci.sh -c -i {{.}}/{{py3_integration_shards_length}} {{/py3_integration_shards}} {{#py2_blacklist_integration_shards}} - - <<: *py2_linux_test_config - name: "Blacklisted integration tests for pants - shard {{.}} (Py2 PEX w/ Py3 constraints)" + - <<: *py27_linux_test_config + name: "Blacklisted integration tests for pants - shard {{.}} (Py2.7 PEX w/ Py3.6 constraints)" stage: *test env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="['CPython>=3.6']" - - CACHE_NAME=integrationshard{{.}}.py2blacklist + - CACHE_NAME=integrationshard{{.}}.py27blacklist script: - ./build-support/bin/travis-ci.sh -c2w -i {{.}}/{{py2_blacklist_integration_shards_length}} {{/py2_blacklist_integration_shards}} {{#cron_integration_shards}} - - <<: *py2_linux_test_config - name: "Integration tests for pants - shard {{.}} (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Integration tests for pants - shard {{.}} (Py2.7 PEX)" env: - - *py2_linux_test_config_env + - *py27_linux_test_config_env - CACHE_NAME=cronshard{{.}} script: - ./build-support/bin/travis-ci.sh -c2 -i {{.}}/{{cron_integration_shards_length}} @@ -648,34 +652,34 @@ matrix: - <<: *linux_rust_tests - <<: *osx_rust_tests - - <<: *py2_linux_test_config - name: "Python contrib tests (Py2 PEX)" + - <<: *py27_linux_test_config + name: "Python contrib tests (Py2.7 PEX)" stage: *test env: - - *py2_linux_test_config_env - - CACHE_NAME=linuxcontribtests.py2 + - *py27_linux_test_config_env + - CACHE_NAME=linuxcontribtests.py27 script: - ./build-support/bin/travis-ci.sh -2n - - <<: *py3_linux_test_config - name: "Python contrib tests (Py3 PEX)" + - <<: *py36_linux_test_config + name: "Python contrib tests (Py3.6 PEX)" env: - - *py3_linux_test_config_env - - CACHE_NAME=linuxcontribtests.py3 + - *py36_linux_test_config_env + - CACHE_NAME=linuxcontribtests.py36 script: - ./build-support/bin/travis-ci.sh -n - - <<: *py2_osx_10_12_sanity_check - - <<: *py3_osx_10_12_sanity_check + - <<: *py27_osx_10_12_sanity_check + - <<: *py36_osx_10_12_sanity_check - - <<: *py2_osx_10_13_sanity_check - - <<: *py3_osx_10_13_sanity_check + - <<: *py27_osx_10_13_sanity_check + - <<: *py36_osx_10_13_sanity_check - - <<: *py2_osx_platform_tests - - <<: *py3_osx_platform_tests + - <<: *py27_osx_platform_tests + - <<: *py36_osx_platform_tests - - <<: *py2_jvm_tests - - <<: *py3_jvm_tests + - <<: *py27_jvm_tests + - <<: *py36_jvm_tests - <<: *deploy_stable_multiplatform_pex - <<: *deploy_unstable_multiplatform_pex diff --git a/build-support/virtualenv b/build-support/virtualenv index 90c34ece340..c87c544634c 100755 --- a/build-support/virtualenv +++ b/build-support/virtualenv @@ -9,13 +9,11 @@ VIRTUALENV_PACKAGE_LOCATION=${VIRTUALENV_PACKAGE_LOCATION:-https://pypi.io/packa REPO_ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && git rev-parse --show-toplevel) source ${REPO_ROOT}/build-support/common.sh -if [[ -z "${PY}" ]]; then - python_bin_name="python2.7"; [ "${PANTS_USE_PYTHON3}" = true ] && python_bin_name="python3" - if which "${python_bin_name}" >/dev/null; then - PY=`which ${python_bin_name}` - else - die "No ${python_bin_name} interpreter found on the path. Python will not work!" - fi +# Locate the binary's path to log to user or die gracefully if not found. +if which "${PY}" >/dev/null; then + PY=`which ${PY}` +else + die "No ${PY} interpreter found on the path. Python will not work!" fi log "Using python at ${PY}" diff --git a/pants b/pants index 79c2006d4e9..72a5d3b7c5e 100755 --- a/pants +++ b/pants @@ -53,6 +53,9 @@ source ${HERE}/build-support/pants_venv # + bootstrap_native_code: Builds target-specific native engine binaries. source ${HERE}/build-support/bin/native/bootstrap_code.sh +# Default to using Python 2.7 if not otherwise specified. +export PY="${PY:-python2.7}" + PANTS_EXE="${HERE}/src/python/pants/bin/pants_loader.py" if [[ ! -z "${WRAPPER_REQUIREMENTS}" ]]; then diff --git a/pants3 b/pants3 index 750b829ad09..4362cf1a77c 100755 --- a/pants3 +++ b/pants3 @@ -3,11 +3,19 @@ # Licensed under the Apache License, Version 2.0 (see LICENSE). # This bootstrap script invokes Pants using a Python 3 interpreter. +# +# To constrain to a specific Python 3 version, such as 3.7, prefix the script with `PY=python3.7`. -# Use Py3 under-the-hood -export PANTS_USE_PYTHON3=true +# Default to using Python 3 if not otherwise specified. +export PY="${PY:-python3}" -# Use Py3 for subprocesses -export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS='["CPython>=3.6,<4"]' +# Set interpreter constraints to exactly match the interpreter used for the venv. +# Note that $PY only impacts which virtualenv we use for the parent Pants process; we must also set +# PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS to constrain spawned subprocesses such as tests. Without +# any interpreter constraints, we get the _Py_Dealloc exception (#6985) as Pants will try to use Python 2 +# for subprocesses. Without the exact constraints we set, we could end up using a different Python 3 +# minor version for subprocesses than we do for Pants. +py_major_minor=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') +export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="['CPython==${py_major_minor}.*']" ./pants "$@" diff --git a/src/rust/engine/src/cffi_build.rs b/src/rust/engine/src/cffi_build.rs index d265e84b20a..c5c4d0c1fd6 100644 --- a/src/rust/engine/src/cffi_build.rs +++ b/src/rust/engine/src/cffi_build.rs @@ -114,7 +114,7 @@ fn main() -> Result<(), CffiBuildError> { // When built with Python 2, it works with both Python 2 and Python 3. // So, we check to see if the under-the-hood interpreter has changed and rebuild the native engine // when needed. - println!("cargo:rerun-if-env-changed=PANTS_USE_PYTHON3"); + println!("cargo:rerun-if-env-changed=PY"); if cfg!(target_os = "linux") { println!("cargo:rustc-link-lib=static=stdc++");