Issues with nodeid consistency when pytest --pyargs is not run from the root directory

[anntzer]

Consider the following (standard) test layout:

quux
└── pkg
    ├── __init__.py
    └── tests
        ├── conftest.py
        ├── __init__.py
        └── test_foo.py

where quux is in PYTHONPATH (e.g., add it directly to PYTHONPATH, or quux could actually be site-packages).

conftest.py is set to report the items nodeids:

def pytest_collection_modifyitems(session, config, items):
    for item in items:
        print(item.nodeid)

and test_foo contains a single test, test_bar

If one runs pytest --pyargs pkg, --pyargs pkg.tests, or pkg.tests.test_foo, from quux, then the nodeid is correctly reported as pkg/tests/test_foo.py::test_bar

If one runs the same commands from quux's parent, then the nodeids are reported as quux/tests/test_foo.py::test_bar -- i.e., relative to cwd rather than to where the tests have been found in the PYTHONPATH.

Finally, if one runs the same commands from a directory that is unrelated (not a parent) of quux, then the nodeids are reported as pkg/tests/test_foo.py::test_bar, tests/test_foo.py::test_bar, and test_foo.py::test_bar respectively -- i.e., relative to the parent of the last specified module.

I think the correct behavior is clearly to always report as relative to where the tests have been found in the PYTHONPATH, i.e. pkg/tests/test_foo.py::test_bar.

Tested from a fresh Py3.6 (Arch Linux repo Python) venv with just pytest 3.6.3 installed.

Seems related to #2775.

[nicoddemus]

Hi,

The nodeids are computed from the rootdir, so the it is behaving as expected. Search the docs for rootdir for a more detailed explanation.

(sorry for the terse response, I'm on the phone)

[RonnyPfannschmidt]

to expand a bit - this difference is there so that tests intended to be found with --pyargs will have the same nodeid both in editable installs and in real installs

[anntzer]

But the nodeids are indeed not the same (this is exactly how I found about this):

#!/bin/bash
set -euo pipefail

envdir=/tmp/tmpenv
rootdir=/tmp/quux
python -mvenv "$envdir"

(
    source "$envdir/bin/activate"
    export PIP_CONFIG_FILE=/dev/null
    
    python -mpip install pytest

    mkdir -p "$rootdir"
    cd "$rootdir"
    mkdir -p pkg/tests
    touch pkg/__init__.py
    touch pkg/tests/__init__.py
    echo 'def test_bar(): pass' > pkg/tests/test_foo.py
    echo 'def pytest_collection_modifyitems(session, config, items): print("NODE:", items[0].nodeid)' \
        > pkg/tests/conftest.py
    cat > setup.py <<EOF
from setuptools import setup, find_packages
setup(name="pkg", packages=find_packages())
EOF

    PYTEST_ADDOPTS= python -mpytest --pyargs pkg | grep NODE
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg.tests | grep NODE

    cd "$rootdir" && python -mpip install .
    cd ~  # unrelated directory, real install
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg | grep NODE
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg.tests | grep NODE
    python -mpip uninstall -y pkg

    cd "$rootdir" && python -mpip install -e .
    cd ..  # parent directory, editable install
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg | grep NODE
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg.tests | grep NODE
    cd ~  # unrelated directory, editable install
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg | grep NODE
    PYTEST_ADDOPTS= python -mpytest --pyargs pkg.tests | grep NODE
)

rm -rf "$envdir"
rm -rf "$rootdir"

The relevant outputs are

NODE: pkg/tests/test_foo.py::test_bar
NODE: pkg/tests/test_foo.py::test_bar

NODE: pkg/tests/test_foo.py::test_bar
NODE: tests/test_foo.py::test_bar

NODE: quux/pkg/tests/test_foo.py::test_bar
NODE: quux/pkg/tests/test_foo.py::test_bar

NODE: pkg/tests/test_foo.py::test_bar
NODE: tests/test_foo.py::test_bar

[RonnyPfannschmidt]

thanks for the reproducer
we can work from that, but it may be a while before someone from the core team can investigate the details