Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bazel hangs indefinitely when OOMing #2946

Closed
damienmg opened this issue May 5, 2017 · 4 comments
Closed

Bazel hangs indefinitely when OOMing #2946

damienmg opened this issue May 5, 2017 · 4 comments
Labels
category: misc > misc P2 We'll consider working on this in future. (Assignee optional) type: bug
Milestone
0.7

Comments

@damienmg
Copy link
Contributor

damienmg commented May 5, 2017

The machine it was running was running out of memory and the result from the test log was just stalling

cat ~/Downloads/test_killed_bazel_server.log 
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
bazel binary is at /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/io_bazel/src/test/shell/bin
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.
Extracting Bazel installation...
.............................................
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.
____Loading package: @bazel_tools//tools/jdk
____Loading package: @bazel_tools//tools/cpp
____Loading package: @local_config_xcode//
____Loading package: @local_jdk//
____Loading package: @local_config_cc//
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.
.
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.
____Loading package: @bazel_tools//tools/cpp
____Loading package: @bazel_tools//tools/jdk
____Loading package: @local_config_xcode//
____Loading package: @local_config_cc//
____Loading package: @local_jdk//
INFO: Reading 'startup' options from /home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/bazelrc: --output_user_root=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800/root --host_javabase=/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/bazel-out/local-fastbuild/bin/src/test/shell/integration/experimental_ui_test.runfiles/local_jdk
INFO: $TEST_TMPDIR defined: output root default is '/home/ci/.cache/bazel/_bazel_ci/29502e488c6c26e35c2694f6f4eb1ffe/bazel-sandbox/5999971806873713228/execroot/linux-x86_64/_tmp/5dc9cbb6bafa5b9915cc309242565800'.

Integration tests for bazel's experimental UI

** test_help_color_nobuild *****************************************************
                                                     [bazel development version]
Usage: bazel <command> <options> ...

Available commands:
  analyze-profile     Analyzes build profile data.
  build               Builds the specified targets.
  canonicalize-flags  Canonicalizes a list of bazel options.
  clean               Removes output files and optionally stops the server.
  coverage            Generates code coverage report for specified test targets.
  dump                Dumps the internal state of the bazel server process.
  fetch               Fetches external repositories that are prerequisites to the targets.
  help                Prints help for commands, or the index.
  info                Displays runtime info about the bazel server.
  license             Prints the license of this software.
  mobile-install      Installs targets to mobile devices.
  query               Executes a dependency graph query.
  run                 Runs the specified target.
  shutdown            Stops the bazel server.
  test                Builds and runs the specified test targets.
  version             Prints version information for bazel.

Getting more help:
  bazel help <command>
                   Prints help and options for <command>.
  bazel help startup_options
                   Options for the JVM hosting bazel.
  bazel help target-syntax
                   Explains the syntax for specifying targets.
  bazel help info-keys
                   Displays a list of keys used by the info command.
PASSED: test_help_color_nobuild

** test_terminal_title *********************************************************
Use --strategy=TestRunner=standalone to disable sandboxing for the failing actions.

We should at least notice something is wrong.
@damienmg damienmg added category: misc > misc P2 We'll consider working on this in future. (Assignee optional) type: bug labels May 5, 2017
@damienmg damienmg added this to the 0.7 milestone May 5, 2017
@damienmg
Copy link
Contributor Author

damienmg commented May 5, 2017

Forgot to add that we observed that here: http://ci.bazel.io/job/bazel-tests/BAZEL_VERSION=HEAD-jdk7,PLATFORM_NAME=linux-x86_64/744/console

We went with @aehlig on the slave to find out there was 40G used by the jenkins agent (so also filed bazelbuild/continuous-integration#68)

@djudd-stripe
Copy link

I have attached a thread dump for what I suspect is the same issue (simultaneously #3645 since we are running in a Docker container and I believe that's why we are OOMing). At a glance it looks like maybe a deadlock.

bazel-threads.dump.txt

@djudd-stripe djudd-stripe mentioned this issue Nov 29, 2017
Closed
@djudd-stripe
Copy link

Nevermind, I think that thread dump was a hang due to something else (calling Bazel from a script inside Bazel). Sorry!

@mafanasyev-tri mafanasyev-tri mentioned this issue Feb 9, 2018
Closed
whilp added a commit to whilp/dotfiles that referenced this issue Oct 7, 2018
@whilp
bazel: limit resources in CI
ae93ca3 
Builds are timing out, eg:

https://travis-ci.org/whilp/dotfiles/builds/438142354

Perhaps this is because bazel's estimates for available resources
aren't super when running in docker.

bazelbuild/bazel#3645
bazelbuild/bazel#2946
@meisterT
Copy link
Member

The old CI doesn't exist anymore, this is no longer reproducible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
category: misc > misc P2 We'll consider working on this in future. (Assignee optional) type: bug
Projects
None yet
Milestone
0.7
Development

No branches or pull requests
3 participants
@meisterT @damienmg @djudd-stripe