Resolution failure because of python-requires?

[pawamoy]

• I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

I published a project (mkdocs-coverage version 0.2.2) on PyPI with this Python requirement:

requires-python = ">=3.6.1,<3.10"

PDM fails to resolve this version 0.2.2 in another project where the Python requirement is:

requires-python = ">=3.6.8"

Unable to find a resolution that satisfies the following requirements:
  mkdocs-coverage==0.2.2 (from project)
Please make sure the package names are correct. If so, you can either loosen the version constraints of these dependencies, or set a narrower `requires-python` range in the pyproject.toml.
Traceback (most recent call last):
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/resolvelib/resolvers.py", line 341, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/resolver/core.py", line 24, in _merge_into_criterion
    identifier, crit = _old_merge_into_criterion(self, requirement, parent)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/resolvelib/resolvers.py", line 173, in _merge_into_criterion
    raise RequirementsConflicted(criterion)
resolvelib.resolvers.RequirementsConflicted: Requirements conflict: <NamedRequirement mkdocs-coverage==0.2.2>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 184, in main
    return Core().main(args)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 145, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/core.py", line 140, in main
    f(options.project, options)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/commands/add.py", line 35, in handle
    actions.do_add(
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/actions.py", line 173, in do_add
    resolved = do_lock(project, strategy, tracked_names, reqs)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/cli/actions.py", line 64, in do_lock
    mapping, dependencies, summaries = resolve(
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/pdm/resolver/core.py", line 150, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/resolvelib/resolvers.py", line 473, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/user/.local/pipx/venvs/pdm/lib/python3.8/site-packages/resolvelib/resolvers.py", line 343, in resolve
    raise ResolutionImpossible(e.criterion.information)
resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=<NamedRequirement mkdocs-coverage==0.2.2>, parent=None)]

I was able to pip install this version so it is definitely available.
Note however that I'm not sure if the issue comes from PDM (but it seems like it is).

Actual behavior

PDM fails to resolve the version 0.2.2 of mkdocs-coverage. It manages to resolve the previous version though, 0.2.1, which was managed by Poetry with this Python requirement: python = "^3.6".

Expected behavior

No resolution failure.

Environment Information

$ pdm info && pdm info --env
PDM version:        1.5.2
Python Interpreter: /home/user/.basher-packages/pyenv/pyenv/shims/python (3.8)
Project Root:       /home/user/dev/project
Project Packages:   /home/user/dev/project/__pypackages__/3.8
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.8",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "3.10.0-862.6.3.el7.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Fri Jun 15 17:57:37 EDT 2018",
  "python_full_version": "3.8.8",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "linux"
}

Could be related to #141.

[frostming]

">=3.6.1,<3.10" is a rather strong constraint, and you should make sure the requires-python in pyproject.toml is a subset of that range. Or specify the requirement like mkdocs-coverage==0.2.2; python_version<'3.10'

[frostming]

Not a bug, close it now, thanks

[pawamoy]

Oh, I see, thank you for your help! I'll remove <3.10 from mkdocs-coverage indeed, and use a marker in the dependent project in the meantime.

[pohlt]

I have a similar issue:
I would like to install a package which says Requires-Python: ==3.7.* in its METADATA.
PySpecSet.is_subset returns an unexpected False, although I specified requires-python = ">=3.7" in pyproject.toml.

Am I doing something wrong with the Python requirements or is there an issue with is_subset?

Ok, after some more thinking, I guess it makes sense: If a limit the required Python version to ==3.7.* it is working as expected.

Is it so unusual for a python package to specify that it is only working with 3.7.* and then it would not be found with the default >= 3.x comparison proposed by pdm?

PDM version: 1.5.3
Python Interpreter: c:/python37/python.exe (3.7)
Project Root: C:/Users/z001y9xs/proj/proton/azure_git/openapps-iflow-app
Project Packages: C:\Users\z001y9xs\proj\proton\azure_git\openapps-iflow-app_pypackages_\3.7
{
"implementation_name": "cpython",
"implementation_version": "3.7.9",
"os_name": "nt",
"platform_machine": "AMD64",
"platform_release": "10",
"platform_system": "Windows",
"platform_version": "10.0.18362",
"python_full_version": "3.7.9",
"platform_python_implementation": "CPython",
"python_version": "3.7",
"sys_platform": "win32"
}

[frostming]

@pohlt Wrong direction, PDM tries to guarantee pyproject's requires-python is a subset of the python constraints of ALL candidates.

A narrow range like Requires-Python: ==3.7.* can't work on all pythons accepted by the pyproject hence won't be picked as a candidate.

[pohlt]

It just seems overly optimistic to me to claim that my (or any) package will work with Python 3.7 and beyond, e.g., 4.x, and 42.x, so for me it would make more sense to limit the upper bound like ==3.7.*. But this optimism seems to be common practice, so I'll use it as well. 👍🏼

Anecdote: pydantic says it's fine with anything >=3.6.1 and almost would not have worked with 3.10. 😜

[pawamoy]

PDM tries to guarantee pyproject's requires-python is a subset of the python constraints of ALL candidates.

Does it make sense to enforce this for development dependencies as well?
For example, I want to use the mkdocs-gen-files plugin to build my documentation, and since it is compatible with Python 3.7+ only, I have to bump my python-requires to >=3.7. It feels wrong, because my package definitely works on Python 3.6, and I do want to officially support Python 3.6. Bumping to 3.7 will furthermore restrict packages depending on mine as a dev-dep along the chain. But mkdocs-gen-files is a dev-dep, it has no influence on the final package I'm publishing.

In short, I want to be able to publish packages for Python 3.6+, while still dev-depending on packages requiring higher versions: I'm fine with using 3.6+ to run all my actions (quality, tests, etc.) except for "build the docs" for which I would switch to 3.7+.

In fact, PDM does that in a sense. You have to install it with 3.7+, but it works for lower versions.

What do you think about relaxing that guarantee for dev-deps @frostming? Is it possible technically?

[frostming]

You can use environment markers:

mkdocs-gen-files; python_version>='3.7'

[pawamoy]

I tried that but then the resolution fails:

% pdm lock
✖ 🔒 Lock failed
Unable to find a resolution that satisfies the following requirements:
  dataclasses>=0.6; python_version < "3.7" (from <Candidate black 20.8b1 from https://pypi.org/simple/black/>)
  dataclasses<0.8,>=0.7; python_version < "3.7" (from <Candidate mkdocs-section-index 0.3.0 from https://pypi.org/simple/mkdocs-section-index/>)
  dataclasses>=0.6; python_version < "3.7" (from <Candidate black 20.8b1 from https://pypi.org/simple/black/>)
  dataclasses<0.8,>=0.7; python_version < "3.7" (from <Candidate mkdocs-section-index 0.3.0 from https://pypi.org/simple/mkdocs-section-index/>)
Please make sure the package names are correct. If so, you can either loosen the version constraints of these dependencies, or set a narrower `requires-python` range in the pyproject.toml.
See /tmp/pdm-lock-uxg6l64_.log for detailed debug log.
[ResolutionImpossible]: [RequirementInformation(requirement=<NamedRequirement dataclasses>=0.6; python_version < "3.7">, parent=<Candidate black 20.8b1 from https://pypi.org/simple/black/>), RequirementInformation(requirement=<NamedRequirement dataclasses<0.8,>=0.7; python_version < "3.7">, parent=<Candidate mkdocs-section-index 0.3.0 from https://pypi.org/simple/mkdocs-section-index/>), RequirementInformation(requirement=<NamedRequirement dataclasses>=0.6; python_version < "3.7">, parent=<Candidate black 20.8b1 from https://pypi.org/simple/black/>), RequirementInformation(requirement=<NamedRequirement dataclasses<0.8,>=0.7; python_version < "3.7">, parent=<Candidate mkdocs-section-index 0.3.0 from https://pypi.org/simple/mkdocs-section-index/>)]

Do you want me to open a new issue?

[frostming]

@pawamoy Yes, you can do it, also can you provide the full pyproject.toml?