pkg_tar ignores Python toolchain

[bmclarnon]

pkg_tar delegates to build_tar.py to construct the output tar file. When that tool is run, the system python3 interpreter is used, regardless of the default Bazel python toolchain.

In most cases, this isn't an issue because python3 is available in the path and build_tar's output isn't affected by the Python version, but it's inconvenient when using remote execution: it requires that python3 is present in the build image -- even if the WORKSPACE registers a hermetic Python toolchain.

I verified the issue exists in 1.0.1 with a pretty old version of Bazel (6.5.0). I'm not sure if any rules_pkg rules besides pkg_tar are affected.

I believe the fix is straightforward: make the pkg_tar rule also depend on the current Python toolchain and include the corresponding runfiles in the ctx.actions.run() call. This should work with both platform and in-build runtimes based on the PyRuntimeInfo documentation.

--- pkg/private/tar/tar.bzl
+++ pkg/private/tar/tar.bzl
@@ -187,16 +187,21 @@ def _pkg_tar_impl(ctx):
+    py3_runtime = ctx.toolchains["@bazel_tools//tools/python:toolchain_type"].py3_runtime
     ctx.actions.run(
         mnemonic = "PackageTar",
         progress_message = "Writing: %s" % output_file.path,
         inputs = inputs,
-        tools = [ctx.executable.compressor] if ctx.executable.compressor else [],
+        tools = (
+            [ctx.executable.compressor] if ctx.executable.compressor else [] +
+            [py3_runtime.files] if py3_runtime.files else []
+        ),
         executable = ctx.executable._build_tar,
         arguments = [args],
         outputs = [output_file],
         env = {
             "LANG": "en_US.UTF-8",
             "LC_CTYPE": "UTF-8",
             "PYTHONIOENCODING": "UTF-8",
             "PYTHONUTF8": "1",
+            "PATH": "$PATH" + (":" + py3_runtime.interpreter.dirname if py3_runtime.interpreter else ""),
         },
         use_default_shell_env = True,
     )
@@ -310,3 +315,4 @@ pkg_tar_impl = rule(
         ),
     },
+    toolchains = ["@bazel_tools//tools/python:toolchain_type"],
 )

I'm happy to send a PR if you agree this is worth fixing. I'm also relatively new to Starlark, so I'm also very open to suggestions. Thanks!

[aiuto]

I'm not going to be able to answer any questions until 1/27 because of a family issue.

…

On Tue, Jan 21, 2025 at 9:03 PM Brett McLarnon ***@***.***> wrote: pkg_tar delegates to build_tar.py <https://github.com/bazelbuild/rules_pkg/blob/main/pkg/private/tar/build_tar.py> to construct the output tar file. When that tool is run, the system python3 interpreter is used, regardless of the default Bazel python toolchain. In most cases, this isn't an issue because python3 is available in the path and build_tar's output isn't affected by the Python version, but it's inconvenient when using remote execution <https://bazel.build/remote/rbe>: it requires that python3 is present in the build image -- even if the WORKSPACE registers a hermetic Python toolchain. I verified the issue exists in 1.0.1 with a pretty old version of Bazel (6.5.0). I'm not sure if any rules_pkg rules besides pkg_tar are affected. I believe the fix is straightforward: make the pkg_tar rule also depend on the current Python toolchain and include the corresponding runfiles in the ctx.actions.run() call. This should work with both platform and in-build runtimes based on the PyRuntimeInfo <https://bazel.build/versions/7.0.0/rules/lib/providers/PyRuntimeInfo> documentation. --- pkg/private/tar/tar.bzl+++ pkg/private/tar/tar.bzl@@ -187,16 +187,21 @@+ py3_runtime = ***@***.***_tools//tools/python:toolchain_type"].py3_runtime ctx.actions.run( mnemonic = "PackageTar", progress_message = "Writing: %s" % output_file.path, inputs = inputs,- tools = [ctx.executable.compressor] if ctx.executable.compressor else [],+ tools = (+ [ctx.executable.compressor] if ctx.executable.compressor else [] ++ [py3_runtime.files] if py3_runtime.files else []+ ), executable = ctx.executable._build_tar, arguments = [args], outputs = [output_file], env = { "LANG": "en_US.UTF-8", "LC_CTYPE": "UTF-8", "PYTHONIOENCODING": "UTF-8", "PYTHONUTF8": "1",+ "PATH": "$PATH" + (":" + py3_runtime.interpreter.dirname if py3_runtime.interpreter else ""), }, use_default_shell_env = True, )@@ -310,3 +315,4 @@ ), },+ toolchains = ***@***.***_tools//tools/python:toolchain_type"], ) I'm happy to send a PR if you agree this is worth fixing. I'm also relatively new to Starlark, so I'm also very open to suggestions. Thanks! — Reply to this email directly, view it on GitHub <#926>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHANK675TNI6MB6P54D2L337DAVCNFSM6AAAAABVT223QOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHAYDGMJYHAYDONI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[bmclarnon]

Of course -- and thanks for the heads up!