pdm installs packages although nox re-uses virtualenv #346

pohlt · 2021-03-27T16:39:59Z

I have searched the issue tracker and believe that this is not a duplicate.

Steps to reproduce

I'm using PDM in combination with nox and enabled nox' ability to re-use virtualenvs.
When I start a linting session (which uses black, isort, mypy, and flake8), nox claims to re-use the virtualenv:

nox -e lint
nox > Running session lint
nox > Re-using existing virtual environment at .nox/lint.
nox > pdm install -s lint
Synchronizing working set with lock file: 14 to add, 0 to update, 0 to remove

  ✔ Install appdirs 1.4.4 successful
  ✔ Install black 20.8b1 successful
  ✔ Install click 7.1.2 successful
  ✔ Install importlib-metadata 3.7.3 successful
  ✔ Install flake8 3.9.0 successful
  ✔ Install mccabe 0.6.1 successful
  ✔ Install isort 5.8.0 successful
  ✔ Install mypy-extensions 0.4.3 successful
  ✔ Install pathspec 0.8.1 successful
  ✔ Install pycodestyle 2.7.0 successful
  ✔ Install pyflakes 2.3.1 successful
  ✔ Install toml 0.10.2 successful
  ✔ Install typing-extensions 3.7.4.3 successful
  ✔ Install zipp 3.4.1 successful
Installing the project as an editable package...
  ✔ Install assertion 0.2.0 successful

🎉 All complete!
nox > mypy --disallow-untyped-defs --disallow-untyped-calls assertion/
Success: no issues found in 3 source files
nox > black --check --diff assertion/ tests/ noxfile.py
All done! ✨ 🍰 ✨
7 files would be left unchanged.
nox > isort --check --diff assertion tests noxfile.py
nox > flake8 assertion/ tests/ noxfile.py
nox > Session lint was successful.

In total, the execution takes about 7 seconds with most time spent by pdm.

Interestingly, the result is different when I explicitly disallow re-using the virtualenv

nox -e lint --no-reuse-existing-virtualenvs
nox > Running session lint
nox > Creating virtual environment (virtualenv) using python in .nox/lint
nox > pdm install -s lint
Synchronizing working set with lock file: 17 to add, 0 to update, 0 to remove

  ✔ Install appdirs 1.4.4 successful
  ✔ Install click 7.1.2 successful
  ✔ Install importlib-metadata 3.7.3 successful
  ✔ Install black 20.8b1 successful
  ✔ Install mccabe 0.6.1 successful
  ✔ Install flake8 3.9.0 successful
  ✔ Install mypy 0.812 successful
  ✔ Install isort 5.8.0 successful
  ✔ Install mypy-extensions 0.4.3 successful
  ✔ Install pathspec 0.8.1 successful
  ✔ Install pycodestyle 2.7.0 successful
  ✔ Install pyflakes 2.3.1 successful
  ✔ Install regex 2021.3.17 successful
  ✔ Install toml 0.10.2 successful
  ✔ Install typed-ast 1.4.2 successful
  ✔ Install typing-extensions 3.7.4.3 successful
  ✔ Install zipp 3.4.1 successful
Installing the project as an editable package...
  ✔ Install assertion 0.2.0 successful

🎉 All complete!
nox > mypy --disallow-untyped-defs --disallow-untyped-calls assertion/
Success: no issues found in 3 source files
nox > black --check --diff assertion/ tests/ noxfile.py
All done! ✨ 🍰 ✨
7 files would be left unchanged.
nox > isort --check --diff assertion tests noxfile.py
nox > flake8 assertion/ tests/ noxfile.py
nox > Session lint was successful.

This takes about 12 seconds, maybe because PDM installed three more packages (regex, mypy, typed-ast).

My expectation would be that nox doesn't even need to run PDM (so it might be a nox issue) if I tell it to re-use the virtualenv.

Environment Information

PDM version:        1.4.4                   
Python Interpreter: /usr/bin/python3 (3.8.8)
Project Root:       /home/tom/proj/assertion
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.8",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.11.6-1-default",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Thu Mar 11 16:11:36 UTC 2021 (7358b30)",
  "python_full_version": "3.8.8",
  "platform_python_implementaiton": "CPython",
  "python_version": "3.8",
  "sys_platform": "linux"
}

frostming · 2021-03-28T00:43:34Z

Did you turn on the venv support by pdm config use_venv true? It works perfectly on my side

pohlt · 2021-03-28T06:54:46Z

Here are my settings:

pdm config                                                                                                                         2039ms  Sa 27 Mär 2021 16:44:51 UTC
Home configuration (/home/tom/.pdm/config.toml):
  auto_global = False
  cache_dir = /home/tom/.cache/pdm
  parallel_install = True
  pypi.json_api = False
  pypi.url = https://pypi.org/simple
  pypi.verify_ssl = True
  python.use_pyenv = True
  strategy.resolve_max_rounds = 1000
  strategy.save = compatible
  strategy.update = reuse
  use_venv = True

Project configuration (/home/tom/proj/assertion/.pdm.toml):
  python.path = /usr/bin/python3

frostming · 2021-03-28T07:44:49Z

It makes no sense. How did you get nox installed and whether there are any env vars begin with PDM_?

pohlt · 2021-03-28T08:16:48Z

Both nox and pdm live in their own pipx virtualenv.
I don't see any env vars with PDM.

But I think, I had a different setup first and switched to pipx later own. I'll test it with a fresh repo.

pohlt · 2021-03-28T20:03:57Z

Ok, I just tried pdm and nox in a newly set up project:

Linux 5.11.6, x86_64, fish shell
pdm installed via pipx as version 1.4.4
nox installed via pipx (version 2020.12.31)

project folder tree:

├── noxfile.py
├── pdm.lock
├── pn.egg-info
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── requires.txt
│   ├── SOURCES.txt
│   └── top_level.txt
├── __pycache__
│   └── noxfile.cpython-38.pyc
├── __pypackages__
│   └── 3.8
│       ├── bin
│       │   ├── cowsay
│       │   └── cowsay-3.8
│       ├── include
│       └── lib
│           ├── cowsay
│           │   ├── __init__.py
│           │   ├── __main__.py
│           │   ├── main.py
│           │   └── __pycache__
│           │       ├── __init__.cpython-38.pyc
│           │       ├── __main__.cpython-38.pyc
│           │       └── main.cpython-38.pyc
│           └── cowsay-3.0.dist-info
│               ├── entry_points.txt
│               ├── LICENSE.txt
│               ├── METADATA
│               ├── RECORD
│               ├── SHARED
│               └── top_level.txt
└── pyproject.toml

pyproject.toml:

[project]
name = "pn"
version = "0.1.0"
description = ""
authors = [{ name = "Thomas Pohl", email = "[email protected]" }]
dependencies = []
dev-dependencies = []
requires-python = ">=3.8"
dynamic = ["classifiers"]
license = { text = "MIT" }

[project.urls]
homepage = ""


[project.optional-dependencies]
mooh = ["cowsay~=3.0"]
[build-system]
requires = ["pdm-pep517"]
build-backend = "pdm.pep517.api"

[tool]
[tool.pdm]

noxfile.py

import os
import nox

os.environ.update({"PDM_IGNORE_SAVED_PYTHON": "1"})


@nox.session
def mooh(session):
    session.run("pdm", "install", "-v", "-s", "mooh", external=True)
    session.run("cowsay", "Mooh!")

When I run nox -e mooh --reuse-existing-virtualenv, I get:

nox > Running session mooh
nox > Re-using existing virtual environment at .nox/mooh.
nox > pdm install -v -s mooh
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

Install cowsay 3.0 successful
Installing the project as an editable package...
Preparing isolated env for PEP 517 build...
Requirement already satisfied: setuptools in ./.nox/mooh/lib/python3.8/site-packages (from -r /tmp/pdm-build-reqs-7fbr7x_z.txt (line 1)) (53.1.0)
running egg_info
writing pn.egg-info/PKG-INFO
writing dependency_links to pn.egg-info/dependency_links.txt
writing requirements to pn.egg-info/requires.txt
writing top-level names to pn.egg-info/top_level.txt
reading manifest file 'pn.egg-info/SOURCES.txt'
writing manifest file 'pn.egg-info/SOURCES.txt'
Preparing isolated env for PEP 517 build...
Requirement already satisfied: setuptools in ./.nox/mooh/lib/python3.8/site-packages (from -r /tmp/pdm-build-reqs-gweuz743.txt (line 1)) (53.1.0)
running develop
running egg_info
writing pn.egg-info/PKG-INFO
writing dependency_links to pn.egg-info/dependency_links.txt
writing requirements to pn.egg-info/requires.txt
writing top-level names to pn.egg-info/top_level.txt
reading manifest file 'pn.egg-info/SOURCES.txt'
writing manifest file 'pn.egg-info/SOURCES.txt'
running build_ext
Creating /home/tom/proj/pn/.nox/mooh/lib/python3.8/site-packages/pn.egg-link (link to .)
pn 0.1.0 is already the active version in easy-install.pth

Installed /home/tom/proj/pn
Install pn 0.1.0 successful

🎉 All complete!
nox > cowsay Mooh!
  _____
< Mooh! >
  =====
          \
           \
             ^__^                             
             (oo)\_______                   
             (__)\       )\/\             
                 ||----w |           
                 ||     ||  
                 
                 
nox > Session mooh was successful.

When running it a second (third, ...) time, I still get the line about pdm installing cowsay, but it's faster than the first run (3.5s vs 2.6s). Is the Installing cowsay 3.0 successful just a red herring?

pohlt · 2021-03-28T20:15:57Z

I just tried the same with two heavier packages to see a larger time difference when actually installing packages.

I added an additional math section in my pyproject.toml:

[project.optional-dependencies]
mooh = [
    "cowsay~=3.0",
]
math = [
    "numpy~=1.20",
    "scipy~=1.6",
]

The first nox -e math --reuse-existing-virtualenv takes about 15s, the second only 3s and pdm says All packages are synced to date, nothing to do.

Why is it behaving as expected for these two packages?

frostming · 2021-03-30T07:00:04Z

I still couldn't reproduce it. A Dockerfile may help

pohlt · 2021-03-30T09:11:36Z

Ok, I'll prepare a Dockerfile.

Apart from that, it would be great to have some more output in the verbose mode why pdm thinks it has to re-install packages.

frostming · 2021-03-30T09:19:55Z

why pdm thinks it has to re-install packages.

Based on the comparison between lock file and the current installed working set. And the latter heavily relies on the selection of the Python environment which is created based on the interpreter. When VIRTUAL_ENV exists, its value will always be used as the interpreter path and the whole virtualenv will be reused. And the env var has the second-highest priority only next to what is saved in .pdm.toml

pdm/pdm/project/core.py

Lines 159 to 174 in fd22d4b

    
           if self.project_config.get("python.path") and not os.getenv( 
        
               "PDM_IGNORE_SAVED_PYTHON" 
        
           ): 
        
               saved_path = self.project_config["python.path"] 
        
               if not os.path.isfile(saved_path): 
        
                   del self.project_config["python.path"] 
        
               else: 
        
                   return saved_path 
        
           if os.name == "nt": 
        
               suffix = ".exe" 
        
               scripts = "Scripts" 
        
           else: 
        
               suffix = "" 
        
               scripts = "bin" 
        
           if config["use_venv"] and os.getenv("VIRTUAL_ENV"): 
        
               return os.path.join(os.getenv("VIRTUAL_ENV"), scripts, f"python{suffix}")

Even when PDM_IGNORE_SAVED_PYTHON is set, VIRTUAL_ENV becomes the highest priority.

pohlt · 2021-03-30T11:18:08Z

With this Dockerfile

# docker build -t pdm_nox .
# docker run --rm -ti pdm_nox 

FROM python:3.8-slim

WORKDIR /root/pn

RUN pip install pipx
RUN pipx install pdm
RUN pipx install nox

ENV PATH="/root/.local/bin:${PATH}"
ADD noxfile.py pyproject.toml ./
RUN pdm config use_venv true

CMD /bin/bash

# nox -e mooh --reuse-existing-virtualenv
# nox -e math --reuse-existing-virtualenv

I was not able to reproduce the issue (I'm using fish locally and bash in the container, but that shouldn't make a difference).

Since the selection of the right environment is crucial, a log output in verbose mode would be super helpful.

frostming · 2021-03-30T11:36:30Z

Yeah, sorry, but can you hook into somewhere around this function before the return statement:

pdm/pdm/project/core.py

Lines 202 to 206 in fd22d4b

    
           @property 
        
           def environment(self) -> Environment: 
        
               if not self._environment: 
        
                   self._environment = self.get_environment() 
        
               return self._environment

to see what the value of self.python_executable is

pohlt · 2021-03-31T09:13:13Z

For some reason, some installed packages are not detected in the WorkingSet.

For the nox venv with numpy and scipy it works:

╭────────────────────────── <class 'pip._vendor.pkg_resources.WorkingSet'> ──────────────────────────╮
│ A collection of active distributions on sys.path (or a similar list)                               │
│                                                                                                    │
│ ╭────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ <pip._vendor.pkg_resources.WorkingSet object at 0x7efd4b93ea90>                                │ │
│ ╰────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                    │
│     by_key = {                                                                                     │
│                  'scipy': scipy 1.6.1 (/home/tom/proj/pn/.nox/math/lib64/python3.8/site-packages), │
│                  'numpy': numpy 1.20.2 (/home/tom/proj/pn/.nox/math/lib64/python3.8/site-packages) │
│              }                                                                                     │
│  callbacks = []                                                                                    │
│    entries = ['/home/tom/proj/pn/.nox/math/lib64/python3.8/site-packages']                         │
│ entry_keys = {'/home/tom/proj/pn/.nox/math/lib64/python3.8/site-packages': ['scipy', 'numpy']}     │
╰────────────────────────────────────────────────────────────────────────────────────────────────────╯

For the nox venv with cowsay it doesn't work:

╭──────────────── <class 'pip._vendor.pkg_resources.WorkingSet'> ────────────────╮
│ A collection of active distributions on sys.path (or a similar list)           │
│                                                                                │
│ ╭────────────────────────────────────────────────────────────────────────────╮ │
│ │ <pip._vendor.pkg_resources.WorkingSet object at 0x7f250de554f0>            │ │
│ ╰────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                │
│     by_key = {}                                                                │
│  callbacks = []                                                                │
│    entries = ['/home/tom/proj/pn/.nox/mooh/lib64/python3.8/site-packages']     │
│ entry_keys = {'/home/tom/proj/pn/.nox/mooh/lib64/python3.8/site-packages': []} │
╰────────────────────────────────────────────────────────────────────────────────╯

And because of that, in Synchronizer.compare_with_working_set, cowsay is added to to_add and installed each time I run this nox session.

If I find the time, I will look into the WorkingSet constructor.

pohlt · 2021-03-31T09:16:25Z

Ah, cowsay does not install anything into lib64 but only into lib. I think the WorkingSet constructor only considers lib64 if I look at the output above, right?

frostming · 2021-03-31T09:24:53Z

Ah, it makes sense now, I have come out with a quick fix #364 where you can try if the issue is fixed.

pohlt · 2021-03-31T09:30:15Z

Works now as expected. Cheers! 🍾

pohlt added the 🐛 bug Something isn't working label Mar 27, 2021

frostming mentioned this issue Mar 31, 2021

Try fixing the platform lib issue #364

Merged

2 tasks

frostming closed this as completed in #364 Mar 31, 2021

Amar1729 mentioned this issue Oct 4, 2023

PDM incorrectly uses top-level virtualenv from within a nox session #2297

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pdm installs packages although nox re-uses virtualenv #346

pdm installs packages although nox re-uses virtualenv #346

pohlt commented Mar 27, 2021

frostming commented Mar 28, 2021 •

edited

Loading

pohlt commented Mar 28, 2021

frostming commented Mar 28, 2021

pohlt commented Mar 28, 2021 •

edited

Loading

pohlt commented Mar 28, 2021

pohlt commented Mar 28, 2021

frostming commented Mar 30, 2021 •

edited

Loading

pohlt commented Mar 30, 2021

frostming commented Mar 30, 2021 •

edited

Loading

pohlt commented Mar 30, 2021

frostming commented Mar 30, 2021 •

edited

Loading

pohlt commented Mar 31, 2021

pohlt commented Mar 31, 2021

frostming commented Mar 31, 2021

pohlt commented Mar 31, 2021

pdm installs packages although nox re-uses virtualenv #346

pdm installs packages although nox re-uses virtualenv #346

Comments

pohlt commented Mar 27, 2021

Steps to reproduce

Environment Information

frostming commented Mar 28, 2021 • edited Loading

pohlt commented Mar 28, 2021

frostming commented Mar 28, 2021

pohlt commented Mar 28, 2021 • edited Loading

pohlt commented Mar 28, 2021

pohlt commented Mar 28, 2021

frostming commented Mar 30, 2021 • edited Loading

pohlt commented Mar 30, 2021

frostming commented Mar 30, 2021 • edited Loading

pohlt commented Mar 30, 2021

frostming commented Mar 30, 2021 • edited Loading

pohlt commented Mar 31, 2021

pohlt commented Mar 31, 2021

frostming commented Mar 31, 2021

pohlt commented Mar 31, 2021

frostming commented Mar 28, 2021 •

edited

Loading

pohlt commented Mar 28, 2021 •

edited

Loading

frostming commented Mar 30, 2021 •

edited

Loading

frostming commented Mar 30, 2021 •

edited

Loading

frostming commented Mar 30, 2021 •

edited

Loading