bazel cannot be used for remote builds

[malt3]

Describe the bug

I'm trying to use bazel_6 from nixpkgs on my laptop and use remote builders (buildbarn).
This fails due to patches that are applied in nixpkgs.

Vanilla Bazel has a hardcoded fallback path for bash (/bin/bash) ¹². This path is used to invoke shell actions (ctx.actions.run_shell). On NixOS, this obviously doesn't work (/bin/bash doesn't exist). This is why Bazel from nixpkgs replaces this hardcoded path with a bash from nixpkgs.

This has the downside of not producing a working fallback for $PATH (bash from nixpkgs will default initialize $PATH as /no-such-path). The workaround for this is to default initialize the path variables somewhere else. Currently, this is done here.

This works fine for local execution, since this is where this function is called. However, this function is never called for remote execution (it is a hack to override things for the local execution environment only). So when using Bazel from nixpkgs locally and enabling remote execution, the shell is set to a bash from nixpkgs during the analysis phase but the $PATH variable is explicitly left unset by many actions, leading to the PATH in bash being /no-such-path and resulting in no such file or directory errors when executing tools in a script.

I think there are two possible solutions:

• when using remote execution, do not replace the hardcoded /bin/bash (this is probably not a great fix since it only works for FHS executors)
• instead of replacing /bin/bash with a regular bash from nixpkgs, add a little helper script that adds shell utils to $PATH. This way, any shell rule created by bazel has a minimal set of tools.
• use --shell_executable=/bin/bash when turning on remote execution. This has the downside of only working when the remote builder has a shell in that path.

Steps To Reproduce

See this for a detailed reproducer: tweag/rules_nixpkgs#439

Expected behavior

Using bazel from nixpkgs should work with remote execution.

Additional context

Using this version of bazel with remote execution is becoming more viable due to https://github.com/tweag/rules_nixpkgs.

Notify maintainers

@mboes
@marsam
@uri-canva
@cbley
@olebedev
@groodt
@aherrmann
@ylecornec

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 6.5.9, NixOS, 23.05 (Stoat), 23.05.20231027.1a3c95e`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.6`
 - nixpkgs: `/nix/store/lqfbzmwwanrvbvgsvxa5jbmjbhm5aygr-source`

Footnotes

1. https://github.com/bazelbuild/bazel/blob/37d95846d3258da81c690d4693f1604f5d28087d/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L168 ↩

2. https://github.com/bazelbuild/bazel/blob/37d95846d3258da81c690d4693f1604f5d28087d/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java#L219-L223 ↩

[uri-canva]

#265041 got reverted, I'll reopen it.

[layus]

Could we also remove the related patch here -> https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/build-managers/bazel/bazel_6/actions_path.patch?rgh-link-date=2023-11-02T15%3A26%3A07Z, or is it still needed for some reason ?

[malt3]

Could we also remove the related patch here -> https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/build-managers/bazel/bazel_6/actions_path.patch?rgh-link-date=2023-11-02T15%3A26%3A07Z, or is it still needed for some reason ?

To the best of my understanding, it should be sufficient to use a prepared bash (with tools in the PATH) and remove the patch you mentioned. Probably still a good idea to test this in practice!