Bazel server sometimes hangs, causing test timeout flakes

[kchodorow]

This is an umbrella bug for "vanilla" bazel invocations hanging and timing out on the CI. This seems to happen to integration tests that run Bazel a bunch of times.

Timed out a couple of times today:

• http://ci.bazel.io/job/Gerrit-bazel-tests/661/BAZEL_VERSION=latest,PLATFORM_NAME=ubuntu_16.04-x86_64/console
• http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=latest,PLATFORM_NAME=linux-x86_64/751/console

[brandjon]

I'm not able to reproduce locally. The test doesn't do much besides start up bazel, so it's hard to see how the test could be at fault as opposed to the infrastructure. I'll kick this back to you for now; let me know if there's a logical next step I should take to debug.

[kchodorow]

bazelbuild/continuous-integration#68 was created because of this.

[kchodorow]

Another timeout: http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=latest,PLATFORM_NAME=linux-x86_64/753/console

[haxorz]

this sounds like an issue that i diagnosed internally.

3e5edaf made it so bazel waits for all processed spawned by sandboxed actions (e.g. genrules, tests) to terminate before it considers the action to be complete. e.g. for a test that leaves behind long-running processes, this can result in test timeouts.

---

according to http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=latest,PLATFORM_NAME=linux-x86_64/753/console, the timeout is in //src/test/shell/bazel:workspace_test

https://github.com/bazelbuild/bazel/blob/8970b43c42197378e93339584d27063d082a512d/src/test/shell/bazel/workspace_test.sh doesn't shutdown the blaze server at the end of each test case. bazel is hard-wired to have a default --max_idle_secs of 15 (

bazel/src/main/cpp/startup_options.cc

Line 71 in 8e04230

max_idle_secs = testing ? 15 : (3 * 3600);

) when it is running inside of a test. therefore, every shell integration test has an additional 15 second latency attached to it (because the outer blaze will have to wait 15 seconds extra). perhaps this is causing the ci.bazel timeouts you've been seeing.

the above is just a guess from the information in this issue. follow up me and/or @philwo for assistance with debugging.

[philwo]

This should no longer happen since May 15, 2017 (more precisely c4f271d), as the suspected culprit was rolled back. Are we still seeing this on CI?

[laszlocsomor]

@philwo :

Are we still seeing this on CI?

Maybe, #3072 could be a dupe, as @kchodorow noticed.

[kchodorow]

Please stop assigning this to me, I'm not working on it.