Build errors with Bazel 3.3.1 on NixOS

[curiousleo]

Describe the bug
bazel build //ibazel fails after a fresh checkout.

Reproduction instructions

$ git rev-parse HEAD
435512a8d6806fb1913ab536685febd25ed0ae8e

$ bazel build //ibazel
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/archive/1.0.3.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
INFO: Repository bazel_gazelle_go_repository_tools instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule go_repository_tools defined at:
  /home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/bazel_gazelle/internal/go_repository_tools.bzl:117:38: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
   list_repository_tools_srcs: env: ‘/home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/go_sdk/bin/go’: No such file or directory
ERROR: /home/leo/Code/bazel-watcher/ibazel/BUILD:30:11: no such package '@com_github_fsnotify_fsnotify//': no such package '@bazel_gazelle_go_repository_config//': no such package '@bazel_gazelle_go_repository_tools//': list_repository_tools_srcs: env: ‘/home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/go_sdk/bin/go’: No such file or directory
 and referenced by '//ibazel:go_default_library'
ERROR: Analysis of target '//ibazel:ibazel' failed; build aborted: Analysis failed
INFO: Elapsed time: 2.717s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
    Fetching @com_github_fsnotify_fsnotify; Restarting.
    Fetching @local_config_cc; fetching
    Fetching @bazel_gazelle_go_repository_config; Restarting.
    Fetching @org_golang_x_tools; fetching

Expected behavior
bazel build //ibazel should succeed.

bazel query --output=build //ibazel

# /home/leo/Code/bazel-watcher/ibazel/BUILD:17:10
go_binary(
  name = "ibazel",
  visibility = ["//visibility:public"],
  generator_name = "ibazel",
  generator_function = "go_binary_macro",
  generator_location = "ibazel/BUILD:17:10",
  embed = ["//ibazel:go_default_library"],
  importpath = "github.com/bazelbuild/bazel-watcher/ibazel",
  pure = "on",
  x_defs = {"main.Version": "{STABLE_GIT_VERSION}"},
)
# Rule go_binary defined at (most recent call last):
#   /home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/io_bazel_rules_go/go/private/rules/binary.bzl:112:20 in <toplevel>
#   /home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/io_bazel_rules_go/go/private/rules/rule.bzl:47:16    in go_rule

Loading: 1 packages loaded

Version (please complete the following information):

• OS: NixOS
• Browser: Firefox (not relevant here, I think)
• ibazel Version 435512a
• Bazel version:

  Build label: 3.3.1- (@non-git)
  Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
  Build time: Tue Jan 1 00:00:00 1980 (315532800)
  Build timestamp: 315532800
  Build timestamp as int: 315532800
  

Additional context
None.

[achew22]

My preference is to start working through the error messages one at a time. Could you sync past #440 and tell me if that fixes the issue?

[curiousleo]

My preference is to start working through the error messages one at a time.

Absolutely, makes sense!

Could you sync past #440 and tell me if that fixes the issue?

Done -- same error AFAICT:

❯ git rev-parse HEAD
da0611dfb7607a86f0b7dfed0f7f05fc591f280d

❯ bazel build //ibazel
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Repository bazel_gazelle_go_repository_tools instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule go_repository_tools defined at:
  /home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/bazel_gazelle/internal/go_repository_tools.bzl:117:38: in <toplevel>
ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
   list_repository_tools_srcs: env: ‘/home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/go_sdk/bin/go’: No such file or directory
ERROR: /home/leo/Code/bazel-watcher/ibazel/BUILD:30:11: no such package '@com_github_fsnotify_fsnotify//': no such package '@bazel_gazelle_go_repository_config//': no such package '@bazel_gazelle_go_repository_tools//': list_repository_tools_srcs: env: ‘/home/leo/.cache/bazel/_bazel_leo/8c987a6a9f368f6645434e4b3cfab6b4/external/go_sdk/bin/go’: No such file or directory
 and referenced by '//ibazel:go_default_library'
ERROR: Analysis of target '//ibazel:ibazel' failed; build aborted: Analysis failed

[achew22]

Hrm... I have NO idea why this would matter, but could you try upgrading to bazel 3.7.0? It's what the CI runs are against. Build label: 3.3.1- (@non-git) is also a bit strange as a build label for bazel, where did that come from?

[curiousleo]

This is the Bazel version that ships with NixOS 20.09: https://github.com/NixOS/nixpkgs/blob/release-20.09/pkgs/development/tools/build-managers/bazel/bazel_3/default.nix#L28

Looking this up, I just found that bazel-watcher 0.13.1 also comes with NixOS! https://github.com/NixOS/nixpkgs/blob/release-20.09/pkgs/development/tools/bazel-watcher/default.nix#L16

Installing bazel-watcher via Nix works for me on NixOS, so my use case is satisfied. The bazel-watcher build script for NixOS does a bunch of stuff, my guess is that those shenanigans could have something to do with the issue I've encountered.

But I now have a way to install bazel-watcher, so I'm happy for you to go ahead and close this issue if you like. Thanks for looking into it!

[achew22]

Hrm... I guess I'm glad that it works but dismayed that it doesn't build consistently. I don't know what a nix marker is, but it sounds... bad? maybe? If there is any work that can be done to upstream those differences I would love a PR and a few words to help me understand why it is significant.

Closing, but please feel free to reopen if you need anything else.