Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ImportWarning when importing a PEP 420 namespace package #1111

Closed
bignose-debian opened this issue Jul 29, 2017 · 19 comments
Closed

ImportWarning when importing a PEP 420 namespace package #1111

bignose-debian opened this issue Jul 29, 2017 · 19 comments

Comments

@bignose-debian
Copy link

bignose-debian commented Jul 29, 2017

The pkg_resources code is raising an ImportWarning when it attempts to import a PEP 420 implicit namespace package:

ImportError: Failed to import test module: test.test_methods
Traceback (most recent call last):
  File "/usr/lib/python3.5/unittest/loader.py", line 428, in _find_test_path
    module = self._get_module_from_name(name)
  File "/usr/lib/python3.5/unittest/loader.py", line 369, in _get_module_from_name
    __import__(name)
  File "/home/bignose/Projects/debian/dput/trunk/test/test_methods.py", line 26, in <module>
    import testscenarios
  File "/usr/lib/python3/dist-packages/testscenarios/__init__.py", line 57, in <module>
    from testscenarios.scenarios import (
  File "/usr/lib/python3/dist-packages/testscenarios/scenarios.py", line 34, in <module>
    from testtools import iterate_tests
  File "/usr/lib/python3/dist-packages/testtools/__init__.py", line 124, in <module>
    from pbr.version import VersionInfo
  File "/usr/lib/python3/dist-packages/pbr/version.py", line 25, in <module>
    import pkg_resources
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3037, in <module>
    @_call_aside
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3021, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in _initialize_master_working_set
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3065, in <genexpr>
    for dist in working_set
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2595, in activate
    declare_namespace(pkg)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2163, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2083, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /usr/lib/python3/dist-packages/keyrings: missing __init__

(To reproduce this, enable the -W error option to python3. I did this to try to catch warnings in my code, assuming that libraries would not cause them :-/)

The “Not importing directory […] missing init” contradicts the explicit specification from PEP 420:

Namespace packages [as specified in this PEP] cannot contain an init.py .

@bignose-debian
Copy link
Author

I am not very familiar with namespace packages, so I could be wrong about where the fault lies. I am sure, though, that there is a fault, and it needs to be resolved to avoid the ImportWarning.

@jdufresne
Copy link
Contributor

I'm seeing this as well. It is causing loads of noise in my logs when Python warnings are enabled. Here is a minimal bash script to demonstrate the error:

#!/bin/bash

rm -rf venv
python3 -m venv venv
venv/bin/pip install sphinxcontrib-websupport
venv/bin/python -Werror -c 'import pkg_resources'

Actual output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2958, in <module>
    @_call_aside
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2944, in _call_aside
    f(*args, **kwargs)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2985, in _initialize_master_working_set
    dist.activate(replace=False)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2521, in activate
    declare_namespace(pkg)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2103, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/jon/warning-test/venv/lib64/python3.5/site-packages/pkg_resources/__init__.py", line 2037, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 445, in _find_module_shim
ImportWarning: Not importing directory /home/jon/warning-test/venv/lib64/python3.5/site-packages/sphinxcontrib: missing __init__

stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 22, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove an unnecessary one.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 22, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove an unnecessary one.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 24, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove an unnecessary one.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 24, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove some unnecessary ones.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 24, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove some unnecessary ones.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 24, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove some unnecessary ones.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 24, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove some unnecessary ones.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
stephenfin added a commit to stephenfin/sphinx that referenced this issue Dec 29, 2017
There's an issue with 'pkgutils' and namespace packages. This has been
reported against setuptools [1], but until this is resolved, we simply
need to live with it. Ensure said warnings are filtered from tox too and
remove some unnecessary ones.

[1] pypa/setuptools#1111

Signed-off-by: Stephen Finucane <[email protected]>
orlnub123 added a commit to orlnub123/ASCII-ify that referenced this issue Sep 5, 2018
The warnings are caused by pyfiglet importing pkg_resources. There's an issue about it here pypa/setuptools#1111.
@kleptog
Copy link

kleptog commented Sep 14, 2018

As noted above, the lack of init.py is fine according to PEP420. The error itself comes from the builtin imp module in python2.7:

/usr/lib/python2.7/pkgutil.py:186: ImportWarning: Not importing directory '/home/kleptog/venv/lib/python2.7/site-packages/zc': missing __init__.py
  file, filename, etc = imp.find_module(subname, path)

So the real fix for this is for python itself to stop complaining (given it just works), or barring that suppress the warning in the package that calls it, which is setuptools.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

One plan that addresses this issue is to replace the vast majority of uses of pkg_resources with stdlib functionality (and backports), such as with importlib_resources, importlib_metadata and entrypoints.

a minimal bash script to demonstrate the error

I tried replicating the minimal script on Python 3.7, but it fails first on another warning:

  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 943, in <module>
    collections.MutableMapping.register(ParseResults)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/collections/__init__.py", line 52, in __getattr__
    DeprecationWarning, stacklevel=2)
DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

I've reported that in #1480.

I tried replicating the issue on Python 3.6 but couldn't:

~ $ python3.6 -m venv venv
~ $ venv/bin/pip install sphinxcontrib-websupport
Collecting sphinxcontrib-websupport
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
~ $ venv/bin/python -c 'import pkg_resources'; echo no problem
no problem

I should note that if you have rwt (and you should), it's easier and cleaner to test with this one liner (and doesn't leave you with ./venv):

~ $ python3.6 -m rwt sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
Collecting sphinxcontrib-websupport
  Using cached https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
no problem

Let's see if we can ascertain why the issue exists in your environment but not in mine.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I upgraded my setuptools and pip and still no issue:

~ $ python3.6 -m pip install -q -U setuptools pip
~ $ python3.6 -m rwt -q sphinxcontrib-websupport -- -Werror -c 'import pkg_resources'; echo no problem
no problem

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I tested on Python 3.5.2 and 3.6.4 and 3.6.6, but none of those exhibit the reported failure. Maybe the issue has gone away?

@bignose-debian
Copy link
Author

The issue still occurs for me, with Python 3.6.6.

Using a slightly enhanced version of @jdufresne's test script:

#! /bin/bash

VENV_DIR="$(realpath ./venv)"

rm -rf "$VENV_DIR"
python3 -m venv "$VENV_DIR"
"$VENV_DIR"/bin/python3 --version
"$VENV_DIR"/bin/python3 -m pip install sphinxcontrib-websupport
"$VENV_DIR"/bin/python3 -W error -c 'import pkg_resources'

The output I get:

$ ./jdufresne.test 
Python 3.6.6
Collecting sphinxcontrib-websupport
  Using cached https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in _initialize_master_working_set
    for dist in working_set
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in <genexpr>
    for dist in working_set
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2639, in activate
    declare_namespace(pkg)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2176, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2096, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory /home/bignose/Projects/python/setuptools/issue-1111/venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

@bignose-debian Would you consider creating a Dockerfile that replicates the issue? With that, I could troubleshoot in the environment where the issue occurs.

@bignose-debian
Copy link
Author

@jaraco wrote:

Would you consider creating a Dockerfile that replicates the issue?

Yes, though I haven't a clue how to do that (nor how to know whether the result would correctly replicate the behaviour).

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I'll take a stab at it.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I was able to replicate thus:

draft $ cat > Dockerfile
from python:3.6
RUN python -m venv venv
RUN venv/bin/python --version
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkg_resources'
draft $ docker build .
Sending build context to Docker daemon  13.04MB
Step 1/5 : from python:3.6
3.6: Pulling from library/python
05d1a5232b46: Pull complete
5cee356eda6b: Pull complete
89d3385f0fd3: Pull complete
80ae6b477848: Pull complete
28bdf9e584cc: Pull complete
dec1a1f0462b: Pull complete
a4670d125615: Pull complete
547b45a875f5: Pull complete
102a0247b454: Pull complete
Digest: sha256:0c3b6384565b8524a82096bee143174a1f07a28cfdc090ca63c040251ab10604
Status: Downloaded newer image for python:3.6
 ---> 4f13b7f2138e
Step 2/5 : RUN python -m venv venv
 ---> Running in 31b6974b096e
Removing intermediate container 31b6974b096e
 ---> 48aa5b2ac9ae
Step 3/5 : RUN venv/bin/python --version
 ---> Running in 46a4c6c2e3a2
Python 3.6.6
Removing intermediate container 46a4c6c2e3a2
 ---> 5d232b495a62
Step 4/5 : RUN venv/bin/python -m pip install sphinxcontrib-websupport
 ---> Running in 134e7c778b19
Collecting sphinxcontrib-websupport
  Downloading https://files.pythonhosted.org/packages/52/69/3c2fbdc3702358c5b34ee25e387b24838597ef099761fc9a42c166796e8f/sphinxcontrib_websupport-1.1.0-py2.py3-none-any.whl
Installing collected packages: sphinxcontrib-websupport
Successfully installed sphinxcontrib-websupport-1.1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 134e7c778b19
 ---> 86e837b01253
Step 5/5 : CMD venv/bin/python -W error -c 'import pkg_resources'
 ---> Running in 0ce64945506a
Removing intermediate container 0ce64945506a
 ---> d90cede4387c
Successfully built d90cede4387c
draft $ docker run d90cede4387c
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3088, in <module>
    @_call_aside
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3072, in _call_aside
    f(*args, **kwargs)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in _initialize_master_working_set
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3116, in <genexpr>
    for dist in working_set
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2639, in activate
    declare_namespace(pkg)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2176, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2096, in _handle_ns
    loader = importer.find_module(packageName)
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory /venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__

@orlnub123
Copy link

Not sure if this will be of any help but the warning seems to occur on any namespace packages present in sys.modules that have a "namespace_packages.txt" file in their metadata.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

So I've confirmed the .pth file is present and that causes sphinxcontrib to exist in sys.modules before any import of pkg_resources, which is good.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

Modules that have namespace_packages.txt in their metadata are those that use setuptools to declare namespace_packages, a deprecated but common technique (and difficult to migrate away from).

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I tried replicating the issue on Python 3.6 but couldn't

Glad I pasted my exact commands, because I started replicating it on my local workstation and couldn't figure out what I'd done wrong. I see now I neglected to include the important -W error when I tried to replicate the issue above.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

I learned that removing the .pth file works around the issue. That explains why the issue didn't replicate when under rwt (because the .pth files aren't honored in that mode).

draft $ rm venv/lib/python*/site-packages/sphinxcontrib_websupport*.pth
draft $ venv/bin/python -W error -c 'import pkg_resources'; echo no problem
no problem

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

Digging deeper, I see why the absence of the .pth file bypasses the issue. It's because in pkg_resources.Distribution.activate, the package is not in sys.modules, so the declare_namespace never gets called.

@jaraco
Copy link
Member

jaraco commented Sep 14, 2018

Removing setuptools from the equation, I can replicate the failure with this Dockerfile:

from python:3.6
RUN python -m venv venv
RUN venv/bin/python -m pip install -U pip setuptools
RUN venv/bin/python -m pip install sphinxcontrib-websupport
CMD venv/bin/python -W error -c 'import pkgutil; importer = pkgutil.get_importer("venv/lib/python3.6/site-packages/"); importer.find_module("sphinxcontrib")'

emits

draft $ docker run -it 07e6f2229b27
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap_external>", line 426, in _find_module_shim
ImportWarning: Not importing directory venv/lib/python3.6/site-packages/sphinxcontrib: missing __init__

@bignose-debian
Copy link
Author

I'm glad you got such good diagnosis. The Dockerfile doesn't help me much because I just get a bunch of permission errors trying to use it. (I don't have Docker set up on any of my machines so I'm probably not best suited for going down that path for diagnosis.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants