go_tool_binary fails under remote execution (mktemp: command not found)

[malt3]

Describe the bug

When using remote execution as described in the tutorial, go builds fail with the following message:

INFO: Remote execution message for GoToolchainBinaryBuild @go_sdk//:builder: Action details (uncached result): http://bb-browser.example.com:80/blobs/sha256/historical_execute_response/8b82519fa711e6ec14f57fb9175fe617bfff0019e4233d866b55292c4964ed67-925/
ERROR: /home/malte/.cache/bazel/_bazel_malte/00fb985034c463ac103394fddf3349bf/external/go_sdk/BUILD:27:15: GoToolchainBinaryBuild external/go_sdk/builder [for tool] failed: (Exit 1): bash failed: error executing command (from target @go_sdk//:builder) 
  (cd /home/malte/.cache/bazel/_bazel_malte/00fb985034c463ac103394fddf3349bf/execroot/constellation && \
  exec env - \
  /nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-C7777A24-ST-894cc94080c3/bin/external/go_sdk/builder -trimpath -ldflags '\'''\'' external/io_bazel_rules_go/go/tools/builders/ar.go external/io_bazel_rules_go/go/tools/builders/asm.go external/io_bazel_rules_go/go/tools/builders/builder.go external/io_bazel_rules_go/go/tools/builders/cgo2.go external/io_bazel_rules_go/go/tools/builders/compilepkg.go external/io_bazel_rules_go/go/tools/builders/cover.go external/io_bazel_rules_go/go/tools/builders/edit.go external/io_bazel_rules_go/go/tools/builders/embedcfg.go external/io_bazel_rules_go/go/tools/builders/env.go external/io_bazel_rules_go/go/tools/builders/filter.go external/io_bazel_rules_go/go/tools/builders/filter_buildid.go external/io_bazel_rules_go/go/tools/builders/flags.go external/io_bazel_rules_go/go/tools/builders/generate_nogo_main.go external/io_bazel_rules_go/go/tools/builders/generate_test_main.go external/io_bazel_rules_go/go/tools/builders/importcfg.go external/io_bazel_rules_go/go/tools/builders/link.go external/io_bazel_rules_go/go/tools/builders/pack.go external/io_bazel_rules_go/go/tools/builders/read.go external/io_bazel_rules_go/go/tools/builders/replicate.go external/io_bazel_rules_go/go/tools/builders/stdlib.go external/io_bazel_rules_go/go/tools/builders/stdliblist.go external/io_bazel_rules_go/go/tools/builders/path.go')
# Configuration: a398af2ac26d49408999f021c57935c95bc395a73b4a2060b75e3c4ccf4292dc
# Execution platform: //:ubuntu-act-22-04-platform
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash: line 1: mktemp: command not found
/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash: line 1: mktemp: command not found
warning: GOPATH set to GOROOT () has no effect
build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined

This is a similar (working) action when using Ubuntu and vanilla Bazel locally:

# Configuration: 7ef5b98b65eac4fc464889e87ad35f3b0e52ab8f56c97ba8e1878ec35a0c80e6
# Execution platform: //:ubuntu-act-22-04-platform
SUBCOMMAND: # @go_sdk//:builder [action 'GoToolchainBinaryBuild external/go_sdk/builder [for tool]', configuration: 65fcd5f445b9bbab7d9041ac9c2abcf79c07f40a07e5e5308062e3224a7b10a8, execution platform: //:ubuntu-act-22-04-platform]
(cd /root/.cache/bazel/_bazel_root/00fb985034c463ac103394fddf3349bf/execroot/constellation && \
  exec env - \
  /bin/bash -c 'GOMAXPROCS=1 GOCACHE=$(mktemp -d) GOPATH=$(mktemp -d) external/go_sdk/bin/go build -o bazel-out/k8-opt-exec-3EDBA836-ST-724252c57a52/bin/external/go_sdk/builder -trimpath -ldflags '\'''\'' external/io_bazel_rules_go/go/tools/builders/ar.go external/io_bazel_rules_go/go/tools/builders/asm.go external/io_bazel_rules_go/go/tools/builders/builder.go external/io_bazel_rules_go/go/tools/builders/cgo2.go external/io_bazel_rules_go/go/tools/builders/compilepkg.go external/io_bazel_rules_go/go/tools/builders/cover.go external/io_bazel_rules_go/go/tools/builders/edit.go external/io_bazel_rules_go/go/tools/builders/embedcfg.go external/io_bazel_rules_go/go/tools/builders/env.go external/io_bazel_rules_go/go/tools/builders/filter.go external/io_bazel_rules_go/go/tools/builders/filter_buildid.go external/io_bazel_rules_go/go/tools/builders/flags.go external/io_bazel_rules_go/go/tools/builders/generate_nogo_main.go external/io_bazel_rules_go/go/tools/builders/generate_test_main.go external/io_bazel_rules_go/go/tools/builders/importcfg.go external/io_bazel_rules_go/go/tools/builders/link.go external/io_bazel_rules_go/go/tools/builders/pack.go external/io_bazel_rules_go/go/tools/builders/read.go external/io_bazel_rules_go/go/tools/builders/replicate.go external/io_bazel_rules_go/go/tools/builders/stdlib.go external/io_bazel_rules_go/go/tools/builders/stdliblist.go external/io_bazel_rules_go/go/tools/builders/path.go')

This seems to be related to the following code in rules_go:
https://github.com/bazelbuild/rules_go/blob/4211c6d32ee475a8fde1cfc91571e7c0bed67af4/go/private/rules/binary.bzl#L462

To Reproduce

• Follow the tutorial
• Use the nixpkgs go toolchain
• Build any go binary

Expected behavior

The go binary should build successfully

Environment

• Local: NixOS 23.04 (amd64), Remote: buildbarn with ubuntu 22.04 act
• Version of the code: latest head (3e21df2)

Additional context

Maybe we can install mktemp via nixpkgs?

[malt3]

I think I fully understand the problem now and can offer a solution.

This behavior only occurs when the local Bazel is coming from nixpkgs.
Bazel from nixpkgs has a set of patches that slightly change the behavior compared to vanilla Bazel.

More specifically, there is this issue: NixOS/nixpkgs#94222

And this set of patches that are relevant:

• override hardcoded path to /bin/bash to point to a bash from nixpkgs
  • overrides this hardcoded path to bash
• default initialize $PATH to a bash and a minimal set of tools (including coreutils which contains mktemp)
• the actual $PATH variable used as a fallback. On my build of Bazel, this contains bash and mktemp with the following store paths:
  • /nix/store/y9gr7abwxvzcpg5g73vhnx1fpssr5frr-coreutils-9.3/bin/mktemp
  • /nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash

What happens?

When using this patched Bazel locally with BuildBarn runners remotely, the following happens (to the best of my understanding):

Analysis phase

During the analysis phase, the patched Bazel evaluates this rule and sets the shell to /nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin/bash ¹².

(When using vanilla Bazel, the shell is instead set to /bin/bash).

https://github.com/bazelbuild/rules_go/blob/4211c6d32ee475a8fde1cfc91571e7c0bed67af4/go/private/rules/binary.bzl#L462-L473

Execution phase (local)

When performing a local build, both store paths (bash and mktemp) are in the local nix store.
run_shell uses the hardcoded bash from the nix store. While the whole env including $PATH would be empty under vanilla Bazel, this patch adds coreutils from the nix store to the path containing from the coreutils nix store path and the action can run successfully.

Execution phase (RBE)

During a remote build, run_shell still uses the hardcoded bash from the nix store. However, the env actually empty this time, since this patch is only applicable to local execution. This means the $PATH variable is empty and the bash from nixpkgs sets the $PATH to /no-such-path, leading to mktemp: command not found.

How to fix this?

Short term, I found that I can set --shell_executable=/bin/bash to ensure ctx.actions.run_shell picks a working shell on the runner. Long term, I think we should make the bash from nixpgks work using one of the approaches below.

We could change this patch to point to a bash that includes coreutils in it's embedded $PATH or get this patch to work on the remote runner.

In any case, we also need to upload bazel with its dependencies to the NFS server.

Footnotes

1. https://github.com/bazelbuild/bazel/blob/37d95846d3258da81c690d4693f1604f5d28087d/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L168 ↩

2. https://github.com/bazelbuild/bazel/blob/37d95846d3258da81c690d4693f1604f5d28087d/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L219-L223 ↩

[aherrmann]

Thank you for raising this issue! I haven't gone through all the details, yet, but a Go related mktemp failure also occurred here: #423 (comment) . I've worked around it in 925f8e4 and opened NixOS/nixpkgs#258608 for an upstream fix.

[malt3]

Thank you for taking a look. While both issues are related to Go and mktemp, they have different root causes.

[Artturin]

Fixed in nixpkgs NixOS/nixpkgs#349589