Use python-version-specific requirements.txt instead of single generic requirements.txt #150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
... containing all top-level (*.in) and concrete requirements.txt, the latter parametrized by python version. We have a bunch of dependencies that are only needed on specific python versions (e.g. backports like dataclasses, importlib_resources, etc). We use conditional environment markers for those in the top-level requirements.in files. pip-compile does not yet support generating a combined requirements.txt file where differences across python versions/platforms/archs are reconciled using environmet markers: jazzband/pip-tools#826 They currently recommend running pip-compile on each targeted python environment generating as many concrete requirements.txt files jazzband/pip-tools#651 So this is what I have done here.
…irements.in files
pip-compile setup.py doesn't support environment markers ('; python_version < 3.7'), whereas it does when compiling from requirements.in files.
So we read top-level dependencies from external *.in files and stick
them in the setup.py install_requires and extras_require keywords.
The 'dev' extra is new. One can now do `pip install -e .[dev]` and have
all the development requirements installed alongside the current module.
This is handy for developers bootstrapping a new venv to work on.
It's a common practice among python developers.
This is handy when any top-level deps change. The shell script is just a shorthand for the longish tox command
tox -e py39-dev will install in editable mode (for when one is in a rush). One can also use this as a way to quickly bootstrap a development virtual environment. E.g. after running the above command, a new venv is created in .tox/py39-dev with all the pinned requirements installed and the current module installed in editable mode, ready for editing source files and running directly the pytest command therein
The generic invocation 'tox -e py' does not work any more, because we now install dependencies using python-version-specific requirements.txt files and we need to know the exact python major.minor version. Thus we set TOXENV environment variable such that invoking 'tox' without additional -e options will run only the specified python version.
anthrotype
force-pushed
the
requirements
branch
from
6bf3409 to
79ffc42
Compare
rsheeter
reviewed
Nov 30, 2020
rsheeter
reviewed
Nov 30, 2020
|
It feels like it approaches being simpler to hand-write the requirements file with the correct version bounds and use CI to detect when you screw up |
rsheeter
approved these changes
Nov 30, 2020
Using tox -l command to list them, so we have a single source of truth for what are the versions that we support. And we only need to change one file when, e.g., python3.10 is out.
otherwise pip-compile will write full absolute paths in the generated requirements.txt..
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We have a bunch of dependencies that are only needed on specific python versions, mostly backports like dataclasses, importlib_resources; others like pytype aren't compatible on latest pythons, etc.
Right now we use conditional environment markers in either setup.py's
install_requiresor indev-requirements.into include/exclude named packages for specific py-versions (e.g.dataclasses; python_version < '3.7'.Then we periodically run
pip-compile(from pip-tools suite) to "freeze" these abstract, top-level dependencies and produce two genericrequirements.txtanddev-requirements.txt.The problem is
pip-compileresolves the environment markers based on the python it is run with, so it produces different results depending on whether it is run on python3.6 or python3.9... That means we often have to go and edit manually the generatedrequirements.txtfile generated from pip-compile, which defeats the purpose of it being "generated".Unfortunately pip-compile does not support generating a combined
requirements.txtfile where differences between python versions/platforms/archs are reconciled through respective environment markers (cf. jazzband/pip-tools#826).Pip-tools devs currently recommend running pip-compile on each targeted python environment, i.e. generating as many concrete requirements.txt files as are the python environments one wishes to support: jazzband/pip-tools#651
A distinct but related issue is that currently pip-compile's support for setup.py is limited in that it ignores environment markers, whereas it does use them when compiling from a
requirements.infile.The following PR attempts to fix thes problems.
requirements.infiles; setup.py is changed to read off them so they stay in sync and there's only one place to edit;tox -e py39will install deps frompython3.9-requirements.txt, whereastox -e py36will install from thepython3.6-requirements.txt, and so forth.requirements/pip-compile-all.shto regenerate all requirements.txt for each supported python, whenever any of the top-level dependencies is added/removed/changed. No more need to tweak them manually.