pull-test-infra-bazel broken

[alvaroaleman]

Related: bazelbuild/continuous-integration#1055

What happened:

pull-test-infra-bazel permanently fails with:

ERROR: /home/prow/go/src/github.com/kubernetes/test-infra/hack/BUILD.bazel:16:1:  failed (Exit 34): UNAVAILABLE: HTTP/2 error code: NO_ERROR

Samples:

• Deprecate plank v1 #19647
• Fix APISnoop Conformance Gate Entrypoint #19655
• CAPZ: add anonymous pull to PR E2E job #19659

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Please provide links to example occurrences, if any:

Anything else we need to know?:

[rifelpet]

My PR was failing with this same issue about 15 hours ago. Retrying it now and it appears to be stuck during a bazel build.

There are other PRs with jobs stuck at different points as well.

[thejoycekung]

Similar issues happening for the periodic-kubernetes-bazel-build-master job - kubernetes/kubernetes#95795

[fejta]

This looks related to RBE. One option is to reduce paralleism. A second is to turn off RBE. Going to try out both approaches.

[spiffxp]

FWIW

pull-test-infra-bazel uses RBE (but perhaps without a remote cache)

• first failing job started 2020-10-21 20:39 UTC (https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/test-infra/19651/pull-test-infra-bazel/1319015417548640256)
• pull-test-infra-bazel runs in the default build cluster (there is no cluster: field)
• the --config=ci flag here...

  test-infra/config/jobs/kubernetes/test-infra/test-infra-presubmits.yaml

  Line 18 in d0a3414

  - --config=ci

• means these bazel settings should be used...

  test-infra/.bazelrc

  Lines 23 to 71 in d0a3414

  	# Note needs an instance name
  	# See --config=remote-fejta for a concrete example
  	# https://github.com/bazelbuild/bazel-toolchains/blob/master/bazelrc/bazel-0.27.0.bazelrc
  	build:remote --jobs=500
  	build:remote --host_javabase=@rbe_default//java:jdk
  	build:remote --javabase=@rbe_default//java:jdk
  	build:remote --host_java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
  	build:remote --java_toolchain=@bazel_tools//tools/jdk:toolchain_hostjdk8
  	build:remote --crosstool_top=@rbe_default//cc:toolchain
  	build:remote --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1
  	build:remote --extra_toolchains=@rbe_default//config:cc-toolchain
  	build:remote --extra_execution_platforms=:rbe_with_network
  	build:remote --host_platform=:rbe_with_network
  	build:remote --platforms=:rbe_with_network
  	build:remote --define=EXECUTOR=remote
  	build:remote --remote_executor=grpcs://remotebuildexecution.googleapis.com
  	build:remote --remote_timeout=3600
  	# Improve cache hit rate
  	build:remote --incompatible_strict_action_env=true
  	# Minimize what is downloaded
  	build:inmemory --experimental_inmemory_jdeps_files
  	build:inmemory --experimental_inmemory_dotd_files
  	# Minimize what is downloaded
  	build:toplevel --config=inmemory
  	build:toplevel --experimental_remote_download_outputs=toplevel
  	build:minimal --config=inmemory
  	build:minimal --experimental_remote_download_outputs=minimal
  	# --google_credentials=some_file.json
  	build:remote --google_default_credentials=true
  	build:remote --config=toplevel
  	run:remote --experimental_remote_download_outputs=all --noexperimental_inmemory_jdeps_files --noexperimental_inmemory_dotd_files
  	# Compose the remote configs with an instance name
  	# A couple examples below:
  	# --config=ci-instance adds the instance name
  	build:ci-instance --remote_instance_name=projects/k8s-prow-builds/instances/default_instance
  	build:trusted-instance --remote_instance_name=projects/k8s-prow/instances/default_instance
  	# Config we want to use in ci
  	build:ci --config=remote --config=ci-instance

• which means bazel ends up using RBE (though it does not have a remote_cache setting)

periodic-kubernetes-bazel-build-master uses RBE (with an RBE-provided remote cache)

• first failing job started 2020-10-21 20:20 UTC (https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/periodic-kubernetes-bazel-build-master/1319010405707681792)
• periodic-kubernetes-bazel-build-master also runs in default
• also sets a --config=ci flag here...

  test-infra/config/jobs/kubernetes/sig-testing/bazel-build-test.yaml

  Line 180 in 3d3b325

  --config=ci \

• which uses these bazel settings... https://github.com/kubernetes/kubernetes/blob/ededd08ba131b727e60f663bd7217fffaaccd448/build/root/.bazelrc#L55-L90
• which means bazel ends up using RBE

pull-kubernetes-bazel-build uses RBE

• first failing job started 2020-10-21 20:28 UTC (https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-bazel-build/1319012327537774592)

periodic-kubernetes-bazel-test-master does not use RBE

• has remained passing
• another difference: runs in k8s-infra-prow-default cluster

[fejta]

Here is the RBE issue to follow: bazelbuild/continuous-integration#1055

[BenTheElder]

still broken, but seems to be a different issue #19667