Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression in relative paths in build dependencies in PDM 2.3.x #1560

Closed
1 task done
lambda opened this issue Dec 8, 2022 · 2 comments · Fixed by #1561
Closed
1 task done

Regression in relative paths in build dependencies in PDM 2.3.x #1560

lambda opened this issue Dec 8, 2022 · 2 comments · Fixed by #1561
Labels
🐛 bug Something isn't working

Comments

@lambda
Copy link

lambda commented Dec 8, 2022 •
edited
Loading

  • I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

We have a package that implements some build plugin support for some custom Git integration at package build time. We have a test case for that plugin, which is a small example package which depends on the outer package to test that we can build it successfully using several different methods of building a package (we previously had regressions in which it would build successfully one way, but not another; so now we test with pdm build, python -m build, python -m build --wheel, etc). It specified this dependency using a relative path; "../../.." (see also note below about whether this is actually intended to work).

As a small minimal example, consider the following directory structure:

outer
├── inner
│   └── pyproject.toml
└── pyproject.toml

With the following contents. outer/pyproject.toml:

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

[project]
name = "outer"
description = "Test for PDM resoluton bug, outer project"
authors = [
    { email = "[email protected]", name = "Beta Technologies" },
]
version = "0.0.0"
requires-python = ">=3.8"

outer/inner/pyproject.toml

[build-system]
requires = ["pdm-pep517", ".."]
build-backend = "pdm.pep517.api"

[project]
name = "inner"
description = "Test for PDM resoluton bug, inner project"
authors = [
    { email = "[email protected]", name = "Beta Technologies" },
]
version = "0.0.0"
requires-python = ">=3.8"

Actual behavior

[brcampbell@beta-0535 inner]$ pdm build -v
Building sdist...
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   outer @ b''
pdm.termui:   pdm-pep517
pdm.termui:   python>=3.11,<3.11.1
pdm.termui:   Adding requirement outer @ b''
Traceback (most recent call last):
  File "/home/brcampbell/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 258, in main
    return Core().main(args)
           ^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 190, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 185, in main
    f(project, options)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/cli/commands/build.py", line 66, in handle
    actions.do_build(
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/cli/actions.py", line 504, in do_build
    loc = SdistBuilder(project.root, project.environment).build(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/builders/sdist.py", line 18, in build
    self.install(self._requires, shared=True)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/builders/base.py", line 298, in install
    install_requirements(missing, env)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/installers/core.py", line 24, in install_requirements
    resolved, _ = resolve(
                  ^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 372, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
         ^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/structs.py", line 136, in __iter__
    self._factory() if self._iterable is None else self._iterable
    ^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/providers.py", line 150, in matches_gen
    candidates = self._find_candidates(reqs[0])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/providers.py", line 127, in _find_candidates
    can = make_candidate(requirement)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/models/candidates.py", line 623, in make_candidate
    return Candidate(req, name, version, link)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/models/candidates.py", line 142, in __init__
    link = req.as_file_link()  # type: ignore
           ^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/models/requirements.py", line 325, in as_file_link
    return Link(url)
           ^^^^^^^^^
  File "<string>", line 7, in __init__
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/unearth/link.py", line 48, in __post_init__
    if self.url.startswith(vcs_prefixes):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: a bytes-like object is required, not 'str'

Expected behavior

This worked correctly in PDM 2.2.1

[brcampbell@beta-0535 inner]$ pipx run --spec pdm==2.2.1 pdm build -v
⚠️  pdm is already on your PATH and installed at /home/brcampbell/.local/bin/pdm. Downloading and running anyway.
Building sdist...
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pdm-pep517
pdm.termui:   outer @ file:///${PROJECT_ROOT}/..
pdm.termui:   python>=3.11,<3.11.1
pdm.termui:   Adding requirement pdm-pep517
pdm.termui:   Adding requirement outer @ file:///${PROJECT_ROOT}/..
pdm.termui:   Adding requirement python>=3.11,<3.11.1
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
pdm.termui: Pinning: outer 0.0.0
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
pdm.termui: Pinning: pdm-pep517 1.0.6
pdm.termui: ======== Ending round 2 ========
pdm.termui: ======== Starting round 3 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: Stable pins:
pdm.termui:       python None
pdm.termui:        outer file:///${PROJECT_ROOT}/..
pdm.termui:   pdm-pep517 1.0.6
pdm.termui: Installing outer 0.0.0
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: Running PEP 517 backend to build a wheel for <Link file:///home/brcampbell/tmp/pdm-path-bug/inner/.. (from None)>
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   pdm-pep517
pdm.termui:   python>=3.11,<3.11.1
pdm.termui:   Adding requirement pdm-pep517
pdm.termui:   Adding requirement python>=3.11,<3.11.1
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
pdm.termui: Pinning: pdm-pep517 1.0.6
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: Stable pins:
pdm.termui:       python None
pdm.termui:   pdm-pep517 1.0.6
pdm.termui: Installing pdm-pep517 1.0.6
unearth: Using cached <Link https://files.pythonhosted.org/packages/71/2a/ae8abd6ebb76edd97e0c3dc3e61913df438c17e44a9aed05fd2073bb800f/pdm_pep517-1.0.6-py3-none-any.whl (from None)>
pdm.termui: /tmp/pdm-build-env-m6xu3jsc-shared/lib64/python3.11/site-packages/pdm/pep517/base.py:384: PDMWarning: 'license' is missing
pdm.termui:   if meta.license_expression:
pdm.termui: /tmp/pdm-build-env-m6xu3jsc-shared/lib64/python3.11/site-packages/pdm/pep517/wheel.py:236: PDMWarning: No license files are matched with glob patterns ['LICENSES/*', 'LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*'].
pdm.termui:   for license_file in self.find_license_files():
pdm.termui:  - Adding outer-0.0.0.dist-info/WHEEL
pdm.termui:  - Adding outer-0.0.0.dist-info/METADATA
pdm.termui: Installing pdm-pep517 1.0.6
unearth: Using cached <Link https://files.pythonhosted.org/packages/71/2a/ae8abd6ebb76edd97e0c3dc3e61913df438c17e44a9aed05fd2073bb800f/pdm_pep517-1.0.6-py3-none-any.whl (from None)>
pdm.termui: /tmp/pdm-build-env-ej1qfih6-shared/lib64/python3.11/site-packages/pdm/pep517/sdist.py:53: PDMWarning: No license files are matched with glob patterns ['LICENSES/*', 'LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*'].
pdm.termui:   super()._find_files_iter(for_sdist), self.find_license_files()
pdm.termui: /tmp/pdm-build-env-ej1qfih6-shared/lib64/python3.11/site-packages/pdm/pep517/base.py:384: PDMWarning: 'license' is missing
pdm.termui:   if meta.license_expression:
pdm.termui:  - Adding pyproject.toml
pdm.termui:  - Adding PKG-INFO
Built sdist at /home/brcampbell/tmp/pdm-path-bug/inner/dist/inner-0.0.0.tar.gz
Building wheel...
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: /tmp/pdm-build-env-ej1qfih6-shared/lib64/python3.11/site-packages/pdm/pep517/base.py:384: PDMWarning: 'license' is missing
pdm.termui:   if meta.license_expression:
pdm.termui: /tmp/pdm-build-env-ej1qfih6-shared/lib64/python3.11/site-packages/pdm/pep517/wheel.py:236: PDMWarning: No license files are matched with glob patterns ['LICENSES/*', 'LICEN[CS]E*', 'COPYING*', 'NOTICE*', 'AUTHORS*'].
pdm.termui:   for license_file in self.find_license_files():
pdm.termui:  - Adding inner-0.0.0.dist-info/WHEEL
pdm.termui:  - Adding inner-0.0.0.dist-info/METADATA
Built wheel at /home/brcampbell/tmp/pdm-path-bug/inner/dist/inner-0.0.0-py3-none-any.whl

PDM 2.2.1 is installed, while 2.3.1 is available.
Please run `pdm self update` to upgrade.
Run `pdm config check_update false` to disable the check.

Environment Information

# Paste the output of `pdm info && pdm info --env` below:
$ pdm info && pdm info --env
PDM version:
  2.3.2
Python Interpreter:
  /home/brcampbell/tmp/pdm-path-bug/inner/.venv/bin/python (3.11)
Project Root:
  /home/brcampbell/tmp/pdm-path-bug/inner
Project Packages:
  None
{
  "implementation_name": "cpython",
  "implementation_version": "3.11.0",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "6.0.9-300.fc37.x86_64",
  "platform_system": "Linux",
  "platform_version": "#1 SMP PREEMPT_DYNAMIC Wed Nov 16 17:36:22 UTC 2022",
  "python_full_version": "3.11.0",
  "platform_python_implementation": "CPython",
  "python_version": "3.11",
  "sys_platform": "linux"
}

Partial Workaround

It appears that substituting the path that this expands to in Python 2.2.1 works if this is a build_system.requires: file:///${PROJECT_ROOT}/.. rather than just ..

However, our actual use case is that we're using Hatchling to build, and we're using the dependency in [tool.hatch.build.hooks.custom], and this workaround fails in that usage:

[tool.hatch.build.hooks.custom]
dependencies = ["file:///${PROJECT_ROOT}/.."]

[brcampbell@beta-0535 inner]$ pdm build -v
Building sdist...
pdm.termui: Preparing isolated env for PEP 517 build...
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   hatchling
pdm.termui:   python>=3.11,<3.11.1
pdm.termui:   Adding requirement hatchling
pdm.termui:   Adding requirement python>=3.11,<3.11.1
pdm.termui: ======== Starting round 0 ========
pdm.termui: Pinning: python None
pdm.termui: ======== Ending round 0 ========
pdm.termui: ======== Starting round 1 ========
pdm.termui:   Adding requirement editables>=0.3(from hatchling 1.11.1)
pdm.termui:   Adding requirement packaging>=21.3(from hatchling 1.11.1)
pdm.termui:   Adding requirement pathspec>=0.10.1(from hatchling 1.11.1)
pdm.termui:   Adding requirement pluggy>=1.0.0(from hatchling 1.11.1)
pdm.termui: Pinning: hatchling 1.11.1
pdm.termui: ======== Ending round 1 ========
pdm.termui: ======== Starting round 2 ========
pdm.termui: Pinning: editables 0.3
pdm.termui: ======== Ending round 2 ========
pdm.termui: ======== Starting round 3 ========
pdm.termui: Pinning: packaging 22.0
pdm.termui: ======== Ending round 3 ========
pdm.termui: ======== Starting round 4 ========
pdm.termui: Pinning: pathspec 0.10.2
pdm.termui: ======== Ending round 4 ========
pdm.termui: ======== Starting round 5 ========
pdm.termui: Pinning: pluggy 1.0.0
pdm.termui: ======== Ending round 5 ========
pdm.termui: ======== Starting round 6 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: Stable pins:
pdm.termui:      python None
pdm.termui:   hatchling 1.11.1
pdm.termui:   editables 0.3
pdm.termui:   packaging 22.0
pdm.termui:    pathspec 0.10.2
pdm.termui:      pluggy 1.0.0
pdm.termui: Installing hatchling 1.11.1
unearth: Using cached <Link https://files.pythonhosted.org/packages/d3/af/8feac1326463c275d431e7331695f24a34b508fa046288bad289ff0967f2/hatchling-1.11.1-py3-none-any.whl (from None)>
pdm.termui: Installing editables 0.3
unearth: Using cached <Link https://files.pythonhosted.org/packages/ef/8c/87276afb1ba3193c4c05be83965a1e69b8a14821ce6d688464071b179383/editables-0.3-py3-none-any.whl (from None)>
pdm.termui: Installing packaging 22.0
unearth: Using cached <Link https://files.pythonhosted.org/packages/8f/7b/42582927d281d7cb035609cd3a543ffac89b74f3f4ee8e1c50914bcb57eb/packaging-22.0-py3-none-any.whl (from None)>
pdm.termui: Installing pathspec 0.10.2
unearth: Using cached <Link https://files.pythonhosted.org/packages/42/79/94b21d5fabb97749ca94590315abe150a750483c87add8543781bcb6cd26/pathspec-0.10.2-py3-none-any.whl (from None)>
pdm.termui: Installing pluggy 1.0.0
unearth: Using cached <Link https://files.pythonhosted.org/packages/9e/01/f38e2ff29715251cf25532b9082a1589ab7e4f571ced434f98d0139336dc/pluggy-1.0.0-py2.py3-none-any.whl (from None)>
pdm.termui: ======== Start resolving requirements ========
pdm.termui:   outer @ file:///${PROJECT_ROOT}/..
pdm.termui:   python>=3.11,<3.11.1
pdm.termui:   Adding requirement outer @ file:///${PROJECT_ROOT}/..
Traceback (most recent call last):
  File "/home/brcampbell/.local/bin/pdm", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 258, in main
    return Core().main(args)
           ^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 190, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/core.py", line 185, in main
    f(project, options)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/cli/commands/build.py", line 66, in handle
    actions.do_build(
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/cli/actions.py", line 504, in do_build
    loc = SdistBuilder(project.root, project.environment).build(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/builders/sdist.py", line 20, in build
    self.install(requires)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/builders/base.py", line 298, in install
    install_requirements(missing, env)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/installers/core.py", line 24, in install_requirements
    resolved, _ = resolve(
                  ^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/core.py", line 35, in resolve
    result = resolver.resolve(requirements, max_rounds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 521, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 372, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/resolvers.py", line 172, in _add_to_criteria
    if not criterion.candidates:
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/providers.py", line 155, in <genexpr>
    and all(self.is_satisfied_by(r, can) for r in reqs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/providers.py", line 155, in <genexpr>
    and all(self.is_satisfied_by(r, can) for r in reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/resolver/providers.py", line 167, in is_satisfied_by
    return not candidate.req.is_named and backend.expand_line(
                                          ^^^^^^^^^^^^^^^^^^^^
  File "/home/brcampbell/.local/pipx/venvs/pdm/lib64/python3.11/site-packages/pdm/models/backends.py", line 104, in expand_line
    return line.format(
           ^^^^^^^^^^^^
KeyError: 'PROJECT_ROOT'

Notes

I am not sure if it is intended that this worked; I wasn't able to find any documentation on paths that were just bare relative path, however I haven't found any docs on ${PROJECT_ROOT} either. As far as I can tell, neither of these are specified in any of the PEPs on dependency specification.
@lambda lambda added the 🐛 bug Something isn't working label Dec 8, 2022
@frostming
Copy link
Collaborator

frostming commented Dec 9, 2022 •
edited
Loading

According to PEP 518:

The [build-system] table is used to store build-related data. Initially only one key of the table will be valid and is mandatory for the table: requires. This key must have a value of a list of strings representing PEP 508 dependencies required to execute the build system

Therefore specifying .. as a requirement is not supported(absolute name @ file:// URL without any variables is supported, though), but due to the implementation details, both build and hatch can build without issue, because they all run pip to install requirements. The requirement specification of pip is a superset of PEP 508.

Due to the above fact, PDM can also support this case because it isn't hard to implement. But I will not say it is a bug of PDM.

however I haven't found any docs on ${PROJECT_ROOT} either

That is PDM backend-specific and can't be recognized by other tools.

@frostming frostming mentioned this issue Dec 9, 2022
Merged
2 tasks
@lambda
Copy link
Author

lambda commented Dec 9, 2022

Thanks for the quick fix! Confirmed that this fixes our issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🐛 bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: support relative paths in build requires pdm-project/pdm
2 participants
@lambda @frostming