Prepare Linux UCS2 shard for upgrade to Centos6 base image in #7064 #7418
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Danny had a great point that trying to clone into the original PYENV_ROOT will fail if it already exists. Instead, we can simply use a new root. This also allows us to keep more of the original code because there is no longer ambiguity within PYENV_ROOT/shims.
If there are already pyenv installed Pythons, then we want to be able to use them. Namely, we're going to be porting some of our build scripts to Python 3 and we need to ensure we have Python 3 on every shard. Overriding it means we could not use the Python 3 installed on centos6. We can point to ~/.pyenv/versions/2.7.15/bin/python to disambiguate.
In normal Travis, we have to check if PYENV_BIN exists because Travis will automatically create the TRAVIS_ROOT folder as a part of caching. Docker, however, does not have caching, so we can simply check if the root folder exists. If we were to stick with PYENV_BIN, we would need to `rm -rf PYENV_ROOT` to make sure the git clone would not fail which is more work than we want.
PATH needs to point to the parent folder of the bin you want included, not its grandparent. It won't look at subfolders, so using the grandparent was causing the PATH entry to not work properly.
I forgot that `pyenv global` will rewrite what `shims/python2.7` points to. If there is already a Python 2 interpreter installed there, `pyenv global` will simply overwrite the shim to point to Python 2.7. This is how Pyenv is supposed to work, and we should not be using the versions/ folder.
This reverts commit 35979e8.
jsirois
approved these changes
Apr 2, 2019
3 tasks
cosmicexplorer
added a commit
that referenced
this pull request
Apr 5, 2019
### Problem We have used a python 3 interpreter in CI for a while now, using `pyenv` to install this in our Dockerfiles instead of relying on CentOS's `scl` features, which don't allow for multiple pythons at a time. Using `pyenv` to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation. That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across `travis_ci` and `travis_ci_py36` Dockerfiles. Using `pyenv` to install python in our `travis_ci` images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase. ### Solution - Use Pyenv to install Python 2.7 and Python 3.6 in the `centos6` Dockerfile. - Remove `travis_cy_py36`. - Modify the `travis_ci` Dockerfile to bootstrap pyenv itself if not available. ### Result We get some CI time back in the bootstrap phase. ### TODO - [ ] merge this PR - [ ] publish the new `centos6` image to dockerhub at `pantsbuild/centos6:latest` - [ ] merge a followup PR removing the manual pyenv bootstrapping from the `travis_ci` image.
stuhood
pushed a commit
that referenced
this pull request
Apr 8, 2019
…7418) ### Problem We will in #7064 start using Pyenv to install Python 2.7 in the base Centos6 image. This will conflict with our Linux UCS2 shard, which currently uses Pyenv to install Python 2.7 with UCS2 instead of the default UCS4. ### Solution Tweak the Dockerfile to avoid any ambiguity with Centos6 image: * Rename the Python version env var to not conflict by marking it as `UCS2`. * Check if `PYENV_ROOT` is already defined. * Check if `PYENV_ROOT` already exists before git cloning to avoid a git failure. The key line is `${PYENV_BIN} global ${PYTHON_27_VERSION_UCS2}`, which will override whatever `${PYENV_ROOT}/shims/python2.7` used to point to.
stuhood
pushed a commit
that referenced
this pull request
Apr 8, 2019
### Problem We have used a python 3 interpreter in CI for a while now, using `pyenv` to install this in our Dockerfiles instead of relying on CentOS's `scl` features, which don't allow for multiple pythons at a time. Using `pyenv` to install Python 2.7 as well instead of relying on the package manager allows us to upgrade or modify the image without screwing up our Python installation. That work began in this PR and ended up in #6981, #7352, #7418, #7483, and more, but it then led to having to duplicate code across `travis_ci` and `travis_ci_py36` Dockerfiles. Using `pyenv` to install python in our `travis_ci` images also involves a significant setup time in bootstrap phases to install our desired pythons. Installing Python in the CentOS 6 base image avoids having to install Python at all in our CI, which saves us minutes in the bootstrap phase. ### Solution - Use Pyenv to install Python 2.7 and Python 3.6 in the `centos6` Dockerfile. - Remove `travis_cy_py36`. - Modify the `travis_ci` Dockerfile to bootstrap pyenv itself if not available. ### Result We get some CI time back in the bootstrap phase. ### TODO - [ ] merge this PR - [ ] publish the new `centos6` image to dockerhub at `pantsbuild/centos6:latest` - [ ] merge a followup PR removing the manual pyenv bootstrapping from the `travis_ci` image.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We will in #7064 start using Pyenv to install Python 2.7 in the base Centos6 image.
This will conflict with our Linux UCS2 shard, which currently uses Pyenv to install Python 2.7 with UCS2 instead of the default UCS4.
Solution
Tweak the Dockerfile to avoid any ambiguity with Centos6 image:
UCS2.PYENV_ROOTis already defined.PYENV_ROOTalready exists before git cloning to avoid a git failure.The key line is
${PYENV_BIN} global ${PYTHON_27_VERSION_UCS2}, which will override whatever${PYENV_ROOT}/shims/python2.7used to point to.