Installing (or syncing) a project with local file dependency fails in docker #721

jokallun · 2021-11-07T09:04:13Z

I have searched the issue tracker and believe that this is not a duplicate.

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

Running sync (or install) in docker for a project with a local dir referenced with ${PROJECT_ROOT} fails
For some reason this only seems to happen in docker (or when running a github action, which maybe uses docker).

Example project in ./app

.
├── local-dependency
│   ├── __init__.py
│   ├── main.py
│   └── pyproject.toml
├── pdm.lock
└── pyproject.toml

pdm.lock

[[package]]
name = "my-lib"
version = "0.1"
path = "./local-dependency"
summary = "UNKNOWN"

[metadata]
lock_version = "3.1"
content_hash = "sha256:d8637a506d94fe132b31d4c60ea5b89ac576d882e005f1244f5626266e150301"

[metadata.files]

pyproject.toml

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Me", email = "[email protected]"},
]
dependencies = [
    "my-lib @ file:///${PROJECT_ROOT}/local-dependency",
]
requires-python = ">=3.9"
dynamic = ["classifiers"]
license = {text = "MIT"}

[project.urls]
homepage = ""

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

[tool]
[tool.pdm]

local-dependency/pyproject.toml

[project]
name = "my_lib"
version = "0.1"
description = ""
authors = [
    {name = "Me", email = "[email protected]"},
]
dependencies = []
requires-python = ">=3.9"
dynamic = ["classifiers"]
license = {text = "MIT"}

[project.urls]
homepage = ""

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

Building the project:

root@8ad215317799:/app# pdm sync -v
Preparing isolated env for PEP 517 build...
Collecting pdm-pep517
  Using cached pdm_pep517-0.8.6-py3-none-any.whl (173 kB)
Installing collected packages: pdm-pep517
Successfully installed pdm-pep517-0.8.6
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 200, in main
    return Core().main(args)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 153, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "/usr/local/lib/python3.9/site-packages/pdm/core.py", line 148, in main
    f(options.project, options)
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/commands/sync.py", line 25, in handle
    actions.do_sync(
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 154, in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
  File "/usr/local/lib/python3.9/site-packages/pdm/cli/actions.py", line 112, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/core.py", line 31, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 472, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 366, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 212, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/usr/local/lib/python3.9/site-packages/resolvelib/resolvers.py", line 202, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/usr/local/lib/python3.9/site-packages/pdm/resolver/providers.py", line 103, in get_dependencies
    deps, requires_python, _ = self.repository.get_dependencies(candidate)
  File "/usr/local/lib/python3.9/site-packages/pdm/models/repositories.py", line 394, in get_dependencies
    reqs, python, summary = super().get_dependencies(candidate)
  File "/usr/local/lib/python3.9/site-packages/pdm/models/repositories.py", line 79, in get_dependencies
    requirements, requires_python, summary = getter(candidate)
  File "/usr/local/lib/python3.9/site-packages/pdm/models/repositories.py", line 386, in _get_dependencies_from_lockfile
    return self.candidate_info[self._identify_candidate(candidate)]
KeyError: ('my-lib', '0.1', 'file:///${PROJECT_ROOT}/local-dependency', False)

I did try this also with non-root user and the result is the same.
Running this fails only in docker. When running the same commands with exactly same files works fine (In windows 10 WSL env).

Environment Information

root@8ad215317799:/app# pdm info
PDM version:        1.10.0
Python Interpreter: /usr/local/bin/python (3.9)
Project Root:       /app
Project Packages:   /app/__pypackages__/3.9

root@8ad215317799:/app# pdm info --env
{
  "implementation_name": "cpython",
  "implementation_version": "3.9.7",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "4.19.128-microsoft-standard",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Tue Jun 23 12:58:10 UTC 2020",
  "python_full_version": "3.9.7",
  "platform_python_implementation": "CPython",
  "python_version": "3.9",
  "sys_platform": "linux"
}

Some thoughts

I did some digging and found that keys in self.candidate_info are (see the last line in stack trace):
('my-lib', '0.1', 'file:///app/local-dependency', False) which then doesn't match the ('my-lib', '0.1', 'file:///${PROJECT_ROOT}/local-dependency', False)

The reason could be here:

pdm/pdm/models/requirements.py

Lines 280 to 298 in ba297eb

    
           def _parse_url(self) -> None: 
        
               if not self.url: 
        
                   if self.path: 
        
                       self.url = path_to_url( 
        
                           self.path.as_posix().replace("${PROJECT_ROOT}", ".") 
        
                       ) 
        
               else: 
        
                   try: 
        
                       self.path = Path( 
        
                           url_to_path( 
        
                               self.url.replace( 
        
                                   "${PROJECT_ROOT}", 
        
                                   Path(".").absolute().as_posix().lstrip("/"), 
        
                               ) 
        
                           ) 
        
                       ) 
        
                   except AssertionError: 
        
                       pass 
        
               self._parse_name_from_url()

The ${PROJECT_ROOT} in self.url is not resolved to local directory (for self.path it is)

Why this only comes up in docker is beyond me.

The text was updated successfully, but these errors were encountered:

jokallun · 2021-11-10T08:55:01Z

The Docker file I used is very plain

FROM python:3.9

WORKDIR /app

RUN python -m pip install -U pdm toml --pre
RUN apt update; apt install -y vim

I mounted the project files when running the container
docker run -it -d -v "/home/jouni/temp/pdmbug":/app

However I cannot now reproduce this as it installs pdm 0.10.1 which work just great (thanks so much for the fix!)
And strangely if I do
python -m pip install -U pdm==0.10.0 toml --pre
I get an error message:

root@4122c1e34baa:/app# pdm
Traceback (most recent call last):
  File "/usr/local/bin/pdm", line 5, in <module>
    from pdm.core import main
  File "/usr/local/lib/python3.8/site-packages/pdm/core.py", line 15, in <module>
    from pdm.installers import Synchronizer
  File "/usr/local/lib/python3.8/site-packages/pdm/installers/__init__.py", line 2, in <module>
    from pdm.installers.installers import Installer
  File "/usr/local/lib/python3.8/site-packages/pdm/installers/installers.py", line 12, in <module>
    from pdm.models.builders import log_subprocessor
  File "/usr/local/lib/python3.8/site-packages/pdm/models/builders.py", line 9, in <module>
    from build.env import IsolatedEnvironment as _Environment
ImportError: cannot import name 'IsolatedEnvironment' from 'build.env' (/usr/local/lib/python3.8/site-packages/build/env.py)

So I can't replicate the original error. I'm bit lost with this.

frostming · 2021-11-10T09:12:22Z

It is 1.10.1

jokallun · 2021-11-10T09:36:43Z

Oh god! That's embarrasing.
Now I could reproduce the error again.
And updating to 1.10.1 makes it work.
Thanks again.

rohit507 · 2022-04-28T05:52:33Z

I've got a series of projects in the same parent directory that seem to cause the same problem. Specifically when I'm trying to move up a directory from PROJECT_ROOT while keeping the pyproject.toml file location independent.

When I use pdm add -e ./../local-dependency the dependency is resolved to an absolute path that won't work when coworkers pull my changes.

Instead I've tried pdm add -e "file:///\${PROJECT_ROOT}/../local-dependency" which proceeds to fail exactly like the case without the .. in this issue.

I have searched the issue tracker and believe that this is not a duplicate.

It's not clear to me that this is separate issue, rather than an edge case of this one.
I could also be going about this the wrong way entirely.

I'm happy to create a new issue if that would help.

(Edit: I have created a new issue at #1059)

Current Setup

Version:

> pdm --version                                                
Python Development Master (PDM), version 1.14.1

Directory:

.
├── local-dependency
│   ├── __init__.py
│   └── pyproject.toml
└── local-dependency-2
    ├── __init__.py
    ├── pdm.lock
    └── pyproject.toml

local-dependency/pyproject.toml:

[project]
name = "my_lib"
version = "0.1"
description = ""
authors = [
    {name = "Me", email = "[email protected]"},
]
dependencies = []
requires-python = ">=3.9"
dynamic = ["classifiers"]
license = {text = "MIT"}

[project.urls]
homepage = ""

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

local-dependency-2/pyproject.toml:

[project]
name = "my_lib_2"
version = "0.1"
description = ""
authors = [
    {name = "Me", email = "[email protected]"},
]
dependencies = [
   "-e file:///${PROJECT_ROOT}/../local-dependency"
]
requires-python = ">=3.9"
dynamic = ["classifiers"]
license = {text = "MIT"}

[project.urls]
homepage = ""

[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

local-dependency-2/pdm.lock:

[[package]]
name = "my-lib"
version = "0.1"
requires_python = ">=3.9"
editable = true
path = "./../local-dependency"
summary = "UNKNOWN"

[metadata]
lock_version = "3.1"
content_hash = "sha256:846ee7de62586775a93ebb56bb357769d9c7f0e955e391656c274c4898200315"

[metadata.files]

command:

> pdm install -v
~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Preparing isolated env for PEP 517 build...
Collecting pdm-pep517
  Using cached pdm_pep517-0.12.3-py3-none-any.whl (302 kB)
Installing collected packages: pdm-pep517
Successfully installed pdm-pep517-0.12.3
/tmp/pdm-build-env-hvtwo5xe-shared/lib/python3.9/site-packages/pdm/pep517/base.py:394: PDMWarning: `classifiers` no longer supports dynamic filling, please remove it from `dynamic` fields and manually supply all the classifiers
  for classifier in meta.classifiers or []:
Traceback (most recent call last):
  File "~/mambaforge/envs/simple-uam/bin/pdm", line 10, in <module>
    sys.exit(main())
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/core.py", line 233, in main
    return Core().main(args)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/core.py", line 168, in main
    raise cast(Exception, err).with_traceback(traceback)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/core.py", line 163, in main
    f(options.project, options)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/cli/commands/install.py", line 49, in handle
    actions.do_sync(
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/cli/actions.py", line 193, in do_sync
    candidates = resolve_candidates_from_lockfile(project, requirements)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/cli/actions.py", line 140, in resolve_candidates_from_lockfile
    mapping, *_ = resolve(
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/resolver/core.py", line 31, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/resolver/providers.py", line 172, in get_dependencies
    deps, requires_python, _ = self.repository.get_dependencies(candidate)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/models/repositories.py", line 407, in get_dependencies
    reqs, python, summary = super().get_dependencies(candidate)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/models/repositories.py", line 70, in get_dependencies
    requirements, requires_python, summary = getter(candidate)
  File "~/mambaforge/envs/simple-uam/lib/python3.9/site-packages/pdm/models/repositories.py", line 399, in _get_dependencies_from_lockfile
    return self.candidate_info[self._identify_candidate(candidate)]
KeyError: ('my-lib', '0.1', 'file:///${PROJECT_ROOT}/../local-dependency', True)

Which looks rather identical to #721 barring the use of ... Thoughts?

jokallun added the 🐛 bug Something isn't working label Nov 7, 2021

frostming mentioned this issue Nov 9, 2021

Fix the URL of requirement when relative path is given #725

Merged

2 tasks

frostming closed this as completed in #725 Nov 9, 2021

frostming added a commit that referenced this issue Nov 9, 2021

Fix the metadata merging, close #721

d813be6

rohit507 mentioned this issue Apr 28, 2022

Installing a project with a path from the parent of PROJECT_ROOT fails with KeyError. #1059

Closed

1 task

dashsatish mentioned this issue May 28, 2022

"pdm sync" fails while building Docker image for an editable dependency #1099

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Installing (or syncing) a project with local file dependency fails in docker #721

Installing (or syncing) a project with local file dependency fails in docker #721

jokallun commented Nov 7, 2021 •

edited by frostming

Loading

jokallun commented Nov 10, 2021

frostming commented Nov 10, 2021 •

edited

Loading

jokallun commented Nov 10, 2021

rohit507 commented Apr 28, 2022 •

edited

Loading

Installing (or syncing) a project with local file dependency fails in docker #721

Installing (or syncing) a project with local file dependency fails in docker #721

Comments

jokallun commented Nov 7, 2021 • edited by frostming Loading

Steps to reproduce

Environment Information

Some thoughts

jokallun commented Nov 10, 2021

frostming commented Nov 10, 2021 • edited Loading

jokallun commented Nov 10, 2021

rohit507 commented Apr 28, 2022 • edited Loading

jokallun commented Nov 7, 2021 •

edited by frostming

Loading

frostming commented Nov 10, 2021 •

edited

Loading

rohit507 commented Apr 28, 2022 •

edited

Loading