Support constraints in python.install in .readthedocs.yml

[webknjaz]

I'm using pip-tools managed constraints files. This means that there's two (in the simplest case) files.
One is requirements.in and the other is requirements.txt. requirements.in would contain unpinned direct dependencies, maybe with a minimum version specified. OTOH requirements.txt contains pins of the whole transitive dependencies tree with file hashes.

Normally, I'd put -c requirements.txt at the top of requirements.in and pip would pick up that argument implicitly.
But I also use dependabot that doesn't support this atm so I'm forced to keep it separate and use both files in pip commands, like pip install -r requirements.in -c requirements.txt.

I think RTD should support setting constraints files, and not only for the requirements install type.

Details

• Read the Docs project URL: N/A
• Build URL (if applicable): N/A
• Read the Docs username (if applicable): N/A

Expected Result

It should be possible to specify the constraints lockfile for pip as in pip install -c constraints.txt.

version: 2
...
python:
  ...
  install:
  - method: pip
    path: pip >= 19.0.3
    constraints:
    - constraints.txt
  ...
  - requirements: requirements.in
    constraints:
    - requirements.txt
    - additional_constraints.txt

Actual Result

It's not possible.

[humitos]

This is another case that can be supported with the proposal made in #8190 using build.jobs.install override instead of adding a python.install.constraints config at Read the Docs core config file.

[astrojuanlu]

Also, @webknjaz could you share a real-world example of the usage you describe? I am not at all sure that pip install -r requirements.in -c requirements.txt is the intended usage for .in files in pip-tools. To my understanding, they help pip-compile produce proper requirements.txt files, and then one can use the .txt ones alone. But I might have misunderstood something.

This issue might not be the best place to discuss it though, but at least it would help us better understand the use case.

[ssbarnea]

The fact that RTD does not allow use of constraints is a real problems and that is the correct way to pin dependencies, not the requirements.txt files.

Now regarding how to fix it, I would likely find more practical to just expose an pip_args options so the users can add entries like -c constrains.txt to it.

Another way to workaround this is to allow people to define environment variables. Defining PIP_CONSTRAINTS=constaints.txt has the same effect on pip calls. Guess what, #6311 is also still open.

[astrojuanlu]

The fact that RTD does not allow use of constraints is a real problems and that is the correct way to pin dependencies, not the requirements.txt files.

Do you have some sources on that @ssbarnea ? It's the first time I read that constraint files, and not requirements files, are "the correct way to pin dependencies".

Yes, setting an env var works, it can be done from the UI.

[humitos]

@webknjaz using the PIP_CONSTRAINT environment variable solves your issue? (see https://pip.pypa.io/en/stable/cli/pip_install/#install-constraint and https://pip.pypa.io/en/stable/topics/configuration/#environment-variables). In that case, you can use the web UI to set this variable.

If that does not work, another approach could be to use build.jobs.post_install (see https://docs.readthedocs.io/en/stable/config-file/v2.html#build-jobs) to run the pip command as you want. However, it's not possible to disable the default pip command that Read the Docs runs for the common/default dependencies; but I think that will be fine anyways.

Please, let me know if any of these potential solutions solve your problem.

[ssbarnea]

The fact that RTD does not allow use of constraints is a real problems and that is the correct way to pin dependencies, not the requirements.txt files.

Do you have some sources on that @ssbarnea ? It's the first time I read that constraint files, and not requirements files, are "the correct way to pin dependencies".

Yes, setting an env var works, it can be done from the UI.

Using the environment variable is valid workaround but it should be documented on the same location we document requirements, so others can do it.

You can find further info about use of constraints at https://github.com/ansible/devtools/wiki/tox#dependency-pinning. We have 10+ projects using this approach and it works quite well, protecting us from external (deps) surprises, while still being able to automate their refresh.

[humitos]

I found this project that is using build.post_install to use the pip --constraint argument: https://github.com/ComPWA/qrules/blob/cfb083f546deab343c14e52abb0f7a96f756b288/.readthedocs.yml#L17-L19

[redeboer]

Yup, thanks for the tip in #7258 (comment), @humitos!
Note that constraint files are committed in this repo and that they constrain all dependencies (also indirect ones) with pip-tools. Environment variables through the web interface won't do here, because these constraint files are updated commit-by-commit.

[adehad]

For reference, the PIP_CONSTAINT env variable method did not work for me. It seems like there is something in the env creaetion process that installs sphinx<2 ...

---
version: 2

formats: all

build:
  os: ubuntu-22.04
  tools:
    python: "3.8"

python:
  install:
    # PIP_CONSTRAINT has been set as env variable in the RTD web interface
    - method: pip
      path: .
      extra_requirements:
        - docs

---

https://readthedocs.org/projects/jira/builds/17557294/

Build log

Read the Docs build information
Build id: 17557294
Project: jira
Version: 1436
Commit: 166ad84ea0e8d2482a2f2e54ad47e28272a773f2
Date: 2022-07-27T21:16:16.001677Z
State: finished
Success: False


[rtd-command-info] start-time: 2022-07-27T21:16:17.799618Z, end-time: 2022-07-27T21:16:18.772890Z, duration: 0, exit-code: 0
git clone --no-single-branch --depth 50 https://github.com/pycontribs/jira .
Cloning into '.'...

[rtd-command-info] start-time: 2022-07-27T21:16:19.101150Z, end-time: 2022-07-27T21:16:19.463008Z, duration: 0, exit-code: 0
git fetch origin --force --tags --prune --prune-tags --depth 50 pull/1436/head:external-1436
From https://github.com/pycontribs/jira
 * [new ref]         refs/pull/1436/head -> external-1436

[rtd-command-info] start-time: 2022-07-27T21:16:19.661328Z, end-time: 2022-07-27T21:16:19.746012Z, duration: 0, exit-code: 0
git checkout --force 166ad84ea0e8d2482a2f2e54ad47e28272a773f2
Note: switching to '166ad84ea0e8d2482a2f2e54ad47e28272a773f2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 166ad84 update sphinx to fix builds and update RTD config due to deprecations

[rtd-command-info] start-time: 2022-07-27T21:16:20.071468Z, end-time: 2022-07-27T21:16:20.135790Z, duration: 0, exit-code: 0
git clean -d -f -f


[rtd-command-info] start-time: 2022-07-27T21:16:25.312149Z, end-time: 2022-07-27T21:16:25.404003Z, duration: 0, exit-code: 0
asdf global python 3.8.13


[rtd-command-info] start-time: 2022-07-27T21:16:25.928466Z, end-time: 2022-07-27T21:16:26.978744Z, duration: 1, exit-code: 0
python -mvirtualenv /home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436
created virtual environment CPython3.8.13.final.0-64 in 713ms
  creator CPython3Posix(dest=/home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/docs/.local/share/virtualenv)
    added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator

[rtd-command-info] start-time: 2022-07-27T21:16:27.297678Z, end-time: 2022-07-27T21:16:34.201861Z, duration: 6, exit-code: 0
/home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436/bin/python -m pip install --upgrade --no-cache-dir pip setuptools<58.3.0
Requirement already satisfied: pip in /home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436/lib/python3.8/site-packages (21.2.3)
Collecting pip
  Downloading pip-22.2.1-py3-none-any.whl (2.0 MB)
Requirement already satisfied: setuptools<58.3.0 in /home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436/lib/python3.8/site-packages (57.4.0)
Collecting setuptools<58.3.0
  Downloading setuptools-58.2.0-py3-none-any.whl (946 kB)
Installing collected packages: setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 57.4.0
    Uninstalling setuptools-57.4.0:
      Successfully uninstalled setuptools-57.4.0
  Attempting uninstall: pip
    Found existing installation: pip 21.2.3
    Uninstalling pip-21.2.3:
      Successfully uninstalled pip-21.2.3
Successfully installed pip-22.2.1 setuptools-58.2.0

[rtd-command-info] start-time: 2022-07-27T21:16:34.383582Z, end-time: 2022-07-27T21:16:37.863465Z, duration: 3, exit-code: 1
/home/docs/checkouts/readthedocs.org/user_builds/jira/envs/1436/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2 jinja2<3.1.0
Collecting mock==1.0.1
  Downloading mock-1.0.1.zip (861 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 861.9/861.9 kB 6.3 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting pillow==5.4.1
  Downloading Pillow-5.4.1.tar.gz (16.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 48.2 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting alabaster!=0.7.5,<0.8,>=0.7
  Downloading alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting commonmark==0.8.1
  Downloading commonmark-0.8.1-py2.py3-none-any.whl (47 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 kB 195.1 MB/s eta 0:00:00
Collecting recommonmark==0.5.0
  Downloading recommonmark-0.5.0-py2.py3-none-any.whl (9.8 kB)
ERROR: Cannot install sphinx<2 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sphinx<2
    The user requested (constraint) sphinx==5.1.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[humitos]

@adehad your project is about ~7 years old and it's not installing the latest version of Sphinx (it's pinned to <2). This was changed on Oct. 20, 2020 to always install the latest available version of Sphinx. However, old projects were not updated to keep compatibility and don't break them.

I updated your project jira to not pin Sphinx<2 now. Can you please try again and let me know if that worked? Thanks!

[adehad]

Thanks @humitos I can confirm that allows the build to work now when I've set the env var.

To be honest I'm still concerned by the command run:
python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx sphinx-rtd-theme readthedocs-sphinx-ext<2.2

As it has locked dependencies that may conflict with our constraints file in the future.

Therefore I will probably still prefer the workaround mentioned earlier
#7258 (comment)

[humitos]

@adehad I understand the concern. We are working towards a way to be able to disable/change the commands Read the Docs execute by default. So, that won't be a limitation on the future

[adehad]

Sounds good, looking forward to seeing that in the future

[humitos]

We already implemented build.jobs config key (read the docs at https://docs.readthedocs.io/en/latest/build-customization.html) that allows people to add constraints when installing packages. Project could use something like the following in the config file:

version: 2
build:
  os: ubuntu-22.04
  tools:
    python: "3.11"
  jobs:
    post_create_environment:
      pip install -e .[doc] -c .constraints/py3.11.txt

Please let me know if this solution works for your use case or not.