ModuleNotFoundError when trying to run pytest

[AlexS12]

• jointly version: master
• Python version: 3.9.7
• Operating System: Ubuntu (inside Windows Subsystem for Linux)

Description

Trying to test the package while doing the review for PyOpenSci (pyOpenSci/software-submission#45) results in different errors

What I Did

• Create a conda environment: conda create -n jointly python poetry
• Clone the project from github.
• poetry install; pre-commit install
• Tried to run all tests with CONTRIBUTING.md instructions py.test tests.test_jointly failed:

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$  py.test tests.test_jointly
================================================= test session starts ==================================================
platform linux -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/asaez/pyopensci/jointly
collected 0 items

================================================ no tests ran in 0.00s =================================================
ERROR: file or directory not found: tests.test_jointly

• Tried to run all tests from tests folder and got this error

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly/tests$ py.test
================================================= test session starts ==================================================
platform linux -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/asaez/pyopensci/jointly/tests
collected 5 items / 4 errors / 1 selected

======================================================== ERRORS ========================================================
___________________________________________ ERROR collecting test_helpers.py ___________________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_helpers.py:20: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
______________________________________ ERROR collecting test_helpers_plotting.py _______________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers_plotting.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_helpers_plotting.py:9: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
_______________________________________ ERROR collecting test_shake_extractor.py _______________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_shake_extractor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_shake_extractor.py:5: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
________________________________________ ERROR collecting test_synchronizer.py _________________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_synchronizer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_synchronizer.py:11: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
=============================================== short test summary info ================================================
ERROR test_helpers.py
ERROR test_helpers_plotting.py
ERROR test_shake_extractor.py
ERROR test_synchronizer.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

• Tried to run the tests from the root folder and got the same error:

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ pytest
================================================= test session starts ==================================================
platform linux -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/asaez/pyopensci/jointly
collected 5 items / 4 errors / 1 selected

======================================================== ERRORS ========================================================
________________________________________ ERROR collecting tests/test_helpers.py ________________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_helpers.py:20: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
___________________________________ ERROR collecting tests/test_helpers_plotting.py ____________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers_plotting.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_helpers_plotting.py:9: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
____________________________________ ERROR collecting tests/test_shake_extractor.py ____________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_shake_extractor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_shake_extractor.py:5: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
_____________________________________ ERROR collecting tests/test_synchronizer.py ______________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_synchronizer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_synchronizer.py:11: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
=============================================== short test summary info ================================================
ERROR tests/test_helpers.py
ERROR tests/test_helpers_plotting.py
ERROR tests/test_shake_extractor.py
ERROR tests/test_synchronizer.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================ 4 errors in 135.08s (0:02:15) =============================================

[AlexS12]

Hi @enra64! I am still experimenting this issue. I am confused about the results.

I will investigate/get some advice to see what's going on. If I cannot get an answer and you confirm that you can run the tests locally I will assume this is a configuration problem in my environment for the review. However, as someone else might hit this same issue, it would be great to have an answer.

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ pytest tests
================================================= test session starts ==================================================
platform linux -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/asaez/pyopensci/jointly
plugins: anyio-3.4.0
collected 5 items / 4 errors / 1 selected

======================================================== ERRORS ========================================================
________________________________________ ERROR collecting tests/test_helpers.py ________________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_helpers.py:20: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
___________________________________ ERROR collecting tests/test_helpers_plotting.py ____________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_helpers_plotting.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_helpers_plotting.py:9: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
____________________________________ ERROR collecting tests/test_shake_extractor.py ____________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_shake_extractor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_shake_extractor.py:5: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
_____________________________________ ERROR collecting tests/test_synchronizer.py ______________________________________
ImportError while importing test module '/home/asaez/pyopensci/jointly/tests/test_synchronizer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../anaconda3/envs/jointly/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_synchronizer.py:11: in <module>
    from tests.parquet_reader import get_parquet_test_data
E   ModuleNotFoundError: No module named 'tests.parquet_reader'
=============================================== short test summary info ================================================
ERROR tests/test_helpers.py
ERROR tests/test_helpers_plotting.py
ERROR tests/test_shake_extractor.py
ERROR tests/test_synchronizer.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================ 4 errors in 133.77s (0:02:13) =============================================

Same error obtained with:

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$  python -m pytest tests
(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly/tests$  python -m pytest
(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ poetry run python -m pytest

Last one should be equivalent to CI config file:

      - name: Test with pytest
        run: |
          poetry run python -m pytest

The solution for me is to add an __init__.py inside tests folder.

(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ touch tests/__init__.py
(jointly) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ pytest tests
================================================= test session starts ==================================================
platform linux -- Python 3.9.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/asaez/pyopensci/jointly
plugins: anyio-3.4.0
collected 27 items

tests/test_examples.py .....                                                                                     [ 18%]
tests/test_helpers.py .........                                                                                  [ 51%]
tests/test_helpers_plotting.py ..                                                                                [ 59%]
tests/test_shake_extractor.py ...                                                                                [ 70%]
tests/test_synchronizer.py ........                                                                              [100%]

============================================ 27 passed in 134.55s (0:02:14) ============================================

However, I cannot understand why I do not obtain the same results in as in your local envioronment or the Github CI (https://github.com/hpi-dhc/jointly/runs/4642228510?check_suite_focus=true)

Run poetry run python -m pytest
  poetry run python -m pytest
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.9/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.9/x64/lib
    VENV: .venv/bin/activate
============================= test session starts ==============================
platform linux -- Python 3.9.9, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/runner/work/jointly/jointly
collected 27 items

tests/test_examples.py .....                                             [ 18%]
tests/test_helpers.py .........                                          [ 51%]
tests/test_helpers_plotting.py ..                                        [ 59%]
tests/test_shake_extractor.py ...                                        [ 70%]
tests/test_synchronizer.py ........                                      [100%]

============================== 27 passed in 7.46s ==============================

[astrojuanlu]

I can't reproduce with venv + poetry, I get the test suite passing normally. conda + poetry is showing some weird behavior on my computer...

$ which poetry
/home/juanlu/.mambaforge/envs/jointly/bin/poetry
$ poetry install
...
  • Updating alabaster (0.7.12 /home/juanlu/.local/lib/python3.9/site-packages -> 0.7.12)  # ????????????????

(not a fan of Poetry in general, so I won't spend more resources debugging this)

@AlexS12 can you try removing your __init__.py workaround and run the following?

>>> import tests
>>> tests
???
>>> tests.__file__
???
>>> import tests.parquet_reader
>>> tests.parquet_reader.__file__
???

I suspect you might have a tests package somewhere else that produces the error you see. This is what I get:

>>> import tests
>>> tests.__file__
>>> tests  # "namespace package" as expected since there is no __init__.py, see PEP 420
<module 'tests' (namespace)>
>>> import tests.parquet_reader
>>> tests.parquet_reader.__file__
'/home/juanlu/Projects/Other/tmp/jointly/tests/parquet_reader.py'

However, if for any reason there is a tests directory somewhere else in your sys.path, it will get imported first. For example:

(.venv) $ mkdir /tmp/tests
(.venv) $ touch /tmp/tests/__init__.py
(.venv) $ export PYTHONPATH=/tmp:$PYTHONPATH
(.venv) $ python
Python 3.8.10 (default, Nov 26 2021, 20:14:08) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tests
>>> tests.__file__  # Woooops
'/tmp/tests/__init__.py'
>>> tests
<module 'tests' from '/tmp/tests/__init__.py'>
>>> import tests.parquet_reader  # This will now fail!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tests.parquet_reader'

[AlexS12]

@astrojuanlu, you hit the nail with that! Thanks!

Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tests
>>> tests.__file__
'/home/asaez/anaconda3/envs/jointly/lib/python3.9/site-packages/tests/__init__.py'
>>> dir(tests)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> import tests.parquet_reader.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'tests.parquet_reader'

Given the content of anaconda3/envs/jointly/lib/python3.9/site-packages/tests/conftest.py, it seems to come from pastel package:

# -*- coding: utf-8 -*-

import pytest

from pastel import Pastel
from pastel.style import Style
from pastel.stack import StyleStack


@pytest.fixture
def pastel():
    return Pastel(True)


@pytest.fixture
def non_decorated_pastel():
    return Pastel(False)


@pytest.fixture
def style():
    return Style()


@pytest.fixture
def stack():
    return StyleStack()

I don't know where the pastel dependency comes from. Given your results, the problem seems to be caused by the way conda installs pastel, but it might be a good idea to avoid the collision somehow. If the __init__.py is not a good idea, what would it be? maybe a conftest.py? Otherwise, it can be noted in the instructions that tests won't run in a conda environment.

[astrojuanlu]

There's two parts of this issue:

1. setup.py installs the tests package sdispater/pastel#7 (almost two years old... unlikely to be fixed?)
2. Tests of jointly doing from tests.xxx import yyy (not a big deal, but pytest encourages using fixtures for that, and other packages like Astropy ship the test utilities)

In any case, I'd say that this is mainly a problem with Pastel. I'm not sure if it's a transitive dependency of jointly?

[AlexS12]

Apparently, pastel is installed when installing poetry with conda:

(base) asaez@ap-5cg1140pq3:~/pyopensci/jointly$ conda create -n jointly-pastel python poetry
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/asaez/anaconda3/envs/jointly-pastel

  added / updated specs:
    - poetry
    - python


The following NEW packages will be INSTALLED:

  ...
  pastel             pkgs/main/noarch::pastel-0.2.1-py_0
  ...

[astrojuanlu]

From what I understand, pastel will no longer be a transitive dependency of Poetry very soon:

python-poetry/poetry#3618

(the reason is that the new Cleo removed its dependency on Clikit python-poetry/cleo#93, which is the one that depended on pastel https://github.com/sdispater/clikit/blob/1da34f16c50a2a7d4a22d21250cb6268c7b3eefd/pyproject.toml#L20)

Because of the temporary nature of the problem, I'd rather not give it more importance 😅

[AlexS12]

Totally agree! Thanks for the support, @astrojuanlu!