Problem loading twisted into autobahn

[chasain]

We have a project that relies on daphne, django and channels, all in python 3.6.

py_binary(
    name = "manage",
    srcs = ["manage.py"],
    deps = [
        pip_install("channels"),
        pip_install("daphne"),
        pip_install("django"),
    ],
)

We're trying out this set of rules since the official rules are still in early stages and getting this error:

File ".../start-dev-django.runfiles/py_pip_deps/pypi__django/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__django/django/core/management/__init__.py", line 377, in execute
    django.setup()
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__django/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__django/django/apps/registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__django/django/apps/config.py", line 116, in create
    mod = import_module(mod_path)
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__channels/channels/apps.py", line 6, in <module>
    import daphne.server
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__daphne/daphne/server.py", line 32, in <module>
    from .ws_protocol import WebSocketFactory
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__daphne/daphne/ws_protocol.py", line 6, in <module>
    from autobahn.twisted.websocket import (
  File ".../start-dev-django.runfiles/py_pip_deps/pypi__autobahn/autobahn/twisted/__init__.py", line 82, in <module>
    __ident__ = 'Autobahn/{}-Twisted/{}-{}/{}'.format(autobahn.__version__, twisted.__version__, platform.python_implementation(), platform.python_version())
AttributeError: module 'twisted' has no attribute '__version__'

It seems to have pulled everything it needs, including pypi__twisted. Any help would be appreciated. Using release 0.1.5

[dillon-giacoppo]

Hi @chasain, can you please post your requirements.txt and a py file that is the minimum reproduction. I'd like to try and reproduce locally.

Also the py_binary example you added is not correct. pip_install goes in the WORKSPACE and the requirement macro is used in the BUILD file.

[chasain]

Ah sorry, I usually change that when I post online, but we use this for improved readability:

load("@py_pip_deps//:requirements.bzl", pip_install = "requirement")

Thanks for looking into this, I'll work on getting you an example this morning.

[chasain]

This seems to work on my end to produce the error, we're on bazel 3.1.0:

settings.py

SECRET_KEY = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
INSTALLED_APPS = [
     'channels',
]

test.py:

#!/usr/bin/env python3
"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

requirements.txt

channels==2.4.0
daphne==2.5.0
django==3.0.4

[thundergolfer]

@chasian this is somewhat tangential, but renaming "requirement" to "pip_install" is a misleading rename. "pip install" is an imperative command suggesting that something is being run to do pip install X when you list pip_install("X"), which actually isn't happening.

requirement("X") is declarative, as in "I, the py_binary, require this package X". It is not the concern of the py_binary how that package is X is installed. That responsibility is left to the repository rules pip_repository and pip_install.

Further, you may be using pip_repository in your project but rules_python_external does provide the pip_install repo rule which means users could be confused into thinking you're calling that function or that you've overloaded the function.

---

Back to the core concern of this issue, thanks for providing the repro 🙂

[dillon-giacoppo]

I have not forgotten about this, but as explained in #52 (comment) the issue is that this depends on rules_python not creating empty __init__.py files automatically.

In the meantime, we can add an option to pip_install which disables namespace conversion from implicit to pkg_util style. This would assume consumers set either legacy_create_init=False or --incompatible_default_to_explicit_init_py.

This suggested workaround should address the issue you are seeing but will require your py_binary targets to disable automatic __init__ file creation.

[dillon-giacoppo]

Fixed with 275995e. To use this workaround, you will need to add enable_implicit_namespace_pkgs = True to your pip_install command and use either legacy_create_init = False on every py_binary and py_target that uses these dependencies, or the global option --incompatible_default_to_explicit_init_py. This is the only workaround we can do without rules_python being patched to not automatically create __init__.py files.

I tested with the example reproduction you gave above and everything works as expected.