rust fails to compile/bootstrap llvm on armv7 due to llvm out of memory error (while there is plenty)

[stefson]

The hardware is a rpi2, it has 971mb of ram available plus 8gb of fast swap via usb hdd. I monitored the build, to be sure - at the point of the failure there is roughly 1.4gb of combined ram and swap used.

this is what happens with -j4: rust-j4-fails.log.gz

sysroot: "/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/build/armv7-unknown-linux-gnueabihf/stage0-sysroot"
libdir: "/var/tmp/portage/dev-lang/rust-1.26.0/work/rust-stage0/lib"
LLVM ERROR: out of memory
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully:

and this error, slightly different one, with -j2: rust-j2-fails.log.gz

sysroot: "/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/build/armv7-unknown-linux-gnueabihf/stage0-sysroot"
libdir: "/var/tmp/portage/dev-lang/rust-1.26.0/work/rust-stage0/lib"
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully:

Last version of rust I built with this machine was rust-1.23.0, with exactly the same setup and -j2, which took 13hrs. Now I'm forced to drop to -j1, which passes, but takes 27hrs.

My dmesg is clean of any errors. Have you built in a check for available ram, which I am failing here, or what is this all about?

[cuviper]

Are you building with debuginfo? See #45854 -- that can exhaust the entire 32-bit address space. You'll get an OOM on that even if there's still swap available, as the process just can't grow any more. I guess if your build still passes with -j1, this may not be your issue.

It's been sufficient for Fedora to just build armv7 without debuginfo, but I haven't looked how much is actually used in that case.

[stefson]

Ah, okay, so you think that this still is an out of memory issue because some process wants to reserve a higher amount of ram than there is theoretically available, due to the limits of the armv7 arch, right?

Well, I know of the fedora patch, but hadn't had the chance yet to understand what it is actually doing. However, I had the idea that it adds rather than removes debuginfo being built? This one #49959

So your solution would be to pass debuginfo-only-std = true via config.toml, and with the patch from fedora debuginfo-tools = false to disable debug info for extended builts? Or is the latter disabled per default - if I don't want extended and don't need debuginfo for them, do I need the patch anyway?

[cuviper]

Yes, in Fedora we're configuring arm with debuginfo-only-std:
https://src.fedoraproject.org/rpms/rust/blob/3212aa76cb0d2c0a16376e7fecdf0e54243518bc/f/rust.spec#_439-445

The patch for debuginfo-tools is not really related. Before that, the extended tools just never got debuginfo in release builds, and now you have the option to enable it. We do have that disabled for arm too, but to be honest, I didn't actually try it enabled. Maybe I'll flip that next time I update. 😃

[stefson]

hmm, apparently there is no kill switch available to disable any debug, too bad. The rust build system is really complex, the gentoo ebuild does severall incarnations of bootstrapping, stage0/1/2, llvm, etc. pp.

where can I read more about all the build options available, and wether they are automatically enabled or disabled on a default config.toml by rust-upstream itself?

--enable-debuginfo --enable-debuginfo-only-std --disable-debuginfo-tools --disable-debuginfo-lines should be more --disable-debuginfo --disable-debuginfo-only-std --disable-debuginfo-tools --disable-debuginfo-lines

[cuviper]

The defaults are shown in config.toml.example.

[cuviper]

For the configuration used by the upstream builds, grep for RUST_CONFIGURE_ARGS in src/ci.

[stefson]

Well, after reading the example file carefully, there shouldn't be any kind of debug enabled per default, and even though the error message comes from a dir called ../build/bootstrap/debug/bootstrap the gentoo ebuild doesn't add the debug values anyway.

Here is the config.toml: config.toml.gz those options should be set, and the rest of the debug values should be disabled as per default in config.toml.example

[alexcrichton]

Note that when LLVM says it ran out of memory I've often seen this connected to an LLVM assert going awry. If LLVM asserts are enabled you'll often get a better error message sooner, and otherwise it's just an assert being compiled out and then the real bug is masked by a seeming "out of memory" message

[stefson]

That means building with the same config.toml but with assertions = true instead of false should give a more precise error message then?

[alexcrichton]

It may!

[stefson]

with assertion = true and -j4, it is the same as before with -j2

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully:

which I think is just another way to state that there is an out-of-memory error, right?
backtrace with export RUST_BACKTRACE=1

expected success, got: exit code: 101
thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1064:9
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
             at /checkout/src/libstd/panicking.rs:537
   6: bootstrap::compile::run_cargo
             at bootstrap/compile.rs:1064
   7: <bootstrap::compile::Rustc as bootstrap::builder::Step>::run
             at bootstrap/compile.rs:496
   8: bootstrap::builder::Builder::ensure
             at bootstrap/builder.rs:841
   9: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at bootstrap/compile.rs:906
  10: bootstrap::builder::Builder::ensure
             at bootstrap/builder.rs:841
  11: bootstrap::builder::Builder::compiler
             at bootstrap/builder.rs:416
  12: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run
             at bootstrap/compile.rs:888
  13: bootstrap::builder::Builder::ensure
             at bootstrap/builder.rs:841
  14: bootstrap::builder::Builder::compiler
             at bootstrap/builder.rs:416
  15: <bootstrap::compile::Std as bootstrap::builder::Step>::make_run
             at bootstrap/compile.rs:57
  16: bootstrap::builder::StepDescription::maybe_run
             at bootstrap/builder.rs:171
  17: bootstrap::builder::StepDescription::run
             at bootstrap/builder.rs:191
  18: bootstrap::builder::Builder::run
             at bootstrap/builder.rs:403
  19: bootstrap::Build::build
             at bootstrap/lib.rs:398
  20: bootstrap::main
             at bootstrap/bin/main.rs:29
  21: std::rt::lang_start::{{closure}}
             at /checkout/src/libstd/rt.rs:74
  22: std::panicking::try::do_call
             at libstd/rt.rs:59
             at libstd/panicking.rs:479
  23: __rust_maybe_catch_panic
             at libpanic_unwind/lib.rs:102
  24: std::rt::lang_start_internal
             at libstd/panicking.rs:458
             at libstd/panic.rs:358
             at libstd/rt.rs:58
  25: std::rt::lang_start
             at /checkout/src/libstd/rt.rs:74
  26: main
  27: __libc_start_main
Traceback (most recent call last):
  File "./x.py", line 20, in <module>
    bootstrap.main()
  File "/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/src/bootstrap/bootstrap.py", line 800, in main
    bootstrap(help_triggered)
  File "/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/src/bootstrap/bootstrap.py", line 791, in bootstrap
    run(args, env=env, verbose=build.verbose)
  File "/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/src/bootstrap/bootstrap.py", line 148, in run
    raise RuntimeError(err)
RuntimeError: failed to run: /var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/build/bootstrap/debug/bootstrap build --verbose --config=/var/tmp/portage/dev-lang/rust-1.26.0/work/rustc-1.26.0-src/config.toml -j4

full build.log.gz

anything helpfull here?