runfiles / PYTHONPATH issue when running a py_binary with proto deps from a custom rule

[illicitonion]

🐞 bug report

Affected Rule

The issue is caused by the rule: py_binary

Is this a regression?

Not as far as I can tell.

Description

A py_binary with proto deps doesn't seem to be trivially runnable as a tool from a custom rule - PYTHONPATH entries seem to end up missing (or possibly runfiles aren't built/propagated properly).

🔬 Minimal Reproduction

https://github.com/illicitonion/repro-bazel-python-proto-import-error

See the README.md there for examples of commands which do and don't work. In particular, directly bazel runing the py_binary works, but wrapping it in a trivial rule to run it does not.

🌍 Your Environment

Operating System:
macOS

Output of bazel version:

5.3.2

Rules_python version:

0.16.1

[mattoberle]

I added a few lines to hello_world.py to print out some information from sys.path:

diff --git a/hello_world_printer/src/hello_world.py b/hello_world_printer/src/hello_world.py
index 3d67841..c9445ae 100644
--- a/hello_world_printer/src/hello_world.py
+++ b/hello_world_printer/src/hello_world.py
@@ -1,3 +1,9 @@
+import os
+import sys
+for p in filter(os.path.isdir, sys.path):
+    print(p)
+    for d in sorted(os.listdir(p)):
+        print(f'  {d}')
 from demo_bazel_issue.hello_world_printer.protos.same_directory_proto_python_pb.hello_world_printer.protos import (
     same_directory_proto_pb2 as same_directory_proto
 )

Then I built the package with --sandbox_debug to preserve the tree.

bazel build //hello_world_printer/src:hello_world_via_custom_rule --sandbox_debug

If you look at the environment generated by the rule, distinct_directory appears 3 times in sys.path.

$SANDBOX/bin/hello_world_printer/src/hello_world.runfiles/demo_bazel_issue/distinct_directory/protos/sub_dependency/sub_dependency_proto_python_pb
  __init__.py
  distinct_directory

$SANDBOX/bin/hello_world_printer/src/hello_world.runfiles/demo_bazel_issue/distinct_directory/protos/distinct_directory_proto_python_pb
  __init__.py
  distinct_directory

$SANDBOX/bin/hello_world_printer/src/hello_world.runfiles/demo_bazel_issue
  distinct_directory
  external
  hello_world_printer

Looking at just the first two cases where distinct_directory appears:

$SANDBOX/bin/hello_world_printer/src/hello_world.runfiles/demo_bazel_issue/distinct_directory/protos/sub_dependency/sub_dependency_proto_python_pb/
├── distinct_directory
│   ├── __init__.py
│   ├── protos
│   │   ├── __init__.py
│   │   └── sub_dependency
│   │       ├── __init__.py
│   │       └── sub_dependency_proto_pb2.py
└── __init__.py

$SANDBOX/bin/hello_world_printer/src/hello_world.runfiles/demo_bazel_issue/distinct_directory/protos/distinct_directory_proto_python_pb
├── distinct_directory
│   ├── __init__.py
│   └── protos
│       ├── distinct_directory_proto_pb2.py
│       └── __init__.py
└── __init__.py

The __init__.py files in distinct_directory do not include a line to turn the package into a "namespace" package like this:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

As a result, only the first distinct_directory is used to resolve imports.

---

If you build hello_world without the custom rule:

bazel build //hello_world_printer/src:hello_world

You'll find that distinct_directory (and subpaths) do not include __init__.py files, turning the package into an "implicit namespace package".

---

More info is available on this issue here:

• Python import errors when loading multiple libraries under same namespace rules-proto-grpc/rules_proto_grpc#145

In your example repo, setting legacy_create_init = False, on the py_binary rule fixes the "ImportError".
The situation may reach back to the native python rules:

• Python import errors when loading multiple libraries under same namespace rules-proto-grpc/rules_proto_grpc#145 (comment)

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[github-actions]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"