llvm rebuilt 2 or more times if using src tarballs

[gyakovlev]

starting with 1.35.0 bootstrap always rebuilds llvm,

git could not determine the LLVM submodule commit hash. Assuming that an LLVM build is necessary.

so for downstream distributions and gentoo users (they build from tarballs) internal llvm copy is compiled at least twice per bootstrap, same for lld if it's enabled.

I believe this commit below is the reason:
#59303

[jonas-schievink]

#59303 (comment) says that only a no-op build should be performed in that case, looks like that isn't correct?

[gyakovlev]

second lld build is a no-op indeed.
but something triggers full llvm rebuild every time

there is another bug here, probably related.

after second rebuild, llvm install prefix somehow gets set to current build directory.
so when I run env DESTDIR="/path/somewhere" ./x.py install -v installer tries to install the full llvm copy to
$DESTDIR/tmp/portage/dev-lang/rust-1.35.0/work/rustc-1.35.0-src/build/x86_64-unknown-linux-gnu/llvm/

/tmp/portage/dev-lang/rust-1.35.0/work/rustc-1.35.0-src/build/x86_64-unknown-linux-gnu/llvm/ is a builddir for LLVM.

.../cmake_install.cmake
4:if(NOT DEFINED CMAKE_INSTALL_PREFIX)
5:  set(CMAKE_INSTALL_PREFIX "/tmp/portage/dev-lang/rust-1.35.0/work/rustc-1.35.0-src/build/x86_64-unknown-linux-gnu/llvm")

^ I can see this CMAKE_INSTALL_PREFIX set all over llvm dir in cmake_install.cmake files after second rebuild.

so probably something mutates some cmake knob and CMAKE_INSTALL_PREFIX, triggering full rebuild and setting wrong installation prefix as a side-effect.

[jonas-schievink]

cc @euclio and @cuviper

[gyakovlev]

just fyi, manually rolling back those 3 commits from 1.35.0 tarball fixed build for me
105692c
975ba58
e1daa36

[ghost]

We had the same problem on FreeBSD during the 1.35.0 update. We've skipped the LLVM rebuild with this hack: https://svnweb.freebsd.org/ports/head/lang/rust/files/patch-src_bootstrap_native.rs?view=markup&pathrev=502416

[federicomenaquintero]

This causes a tarball build, which normally takes about 11000 seconds for Suse's SLE, to take over 30000 seconds.

[cuviper]

Do you have any details how you're configuring the build, or how to reproduce this?

I just tried the 1.37.0 tarball with the default config, and it only built LLVM once. I ran the build again and it tried LLVM again, but that was indeed a no-op build, only 6 seconds total. I also tried with --enable-ninja and got the same thing, except the second no-op build was only 3 seconds.

[gyakovlev]

@cuviper still patching 1.37, but I'll check it without patches and will attach my toml file.

[gyakovlev]

[llvm]
optimize = true
release-debuginfo = false
assertions = false
targets = "AMDGPU;PowerPC"
experimental-targets = ""
link-shared = false
[build]
build = "powerpc64le-unknown-linux-gnu"
host = ["powerpc64le-unknown-linux-gnu"]
target = ["powerpc64le-unknown-linux-gnu"]
cargo = "/var/tmp/portage/dev-lang/rust-1.37.0/work/rust-stage0/bin/cargo"
rustc = "/var/tmp/portage/dev-lang/rust-1.37.0/work/rust-stage0/bin/rustc"
docs = false
submodules = false
python = "python3.6"
locked-deps = true
vendor = true
extended = true
tools = ["rustfmt","rls","analysis","src","clippy","cargo",]
verbose = 2
[install]
prefix = "/usr"
libdir = "lib64/rust-1.37.0"
docdir = "share/doc/rust-1.37.0"
mandir = "share/rust-1.37.0/man"
[rust]
optimize = true
debug = false
debug-assertions = false
default-linker = "powerpc64le-unknown-linux-gnu-gcc"
channel = "stable"
rpath = false
lld = false
[target.powerpc64le-unknown-linux-gnu]
cc = "powerpc64le-unknown-linux-gnu-gcc"
cxx = "powerpc64le-unknown-linux-gnu-g++"
linker = "powerpc64le-unknown-linux-gnu-gcc"
ar = "powerpc64le-unknown-linux-gnu-ar"

powerpc example, but it also happens on x86_64

looks like the second build happens on installation.

the build is run using

      env $(cat "config.env)\
          python ./x.py build -vv --config=config.toml -j<somenumber> \
          --exclude src/tools/miri

contents of config.env: CFLAGS_powerpc64le-unknown-linux-gnu=-m64

and the installation command is

 env DESTDIR="/some/staging/path" python ./x.py install -vv --config=config.toml \
      --exclude src/tools/miri || die

and the second rebuild actually happend while running x.py install

I think it does not like DESTDIR and somehow decides to rebuild llvm. i tried to explain in #61206 (comment)
destdir is not a final destination, it's a temporary staging location, where package manager takes over and packs it and later installs onto filesystem. it's critical for distributions.

full build logs attached. it's huge, some key points

line 1416 - first llvm build
line 17259 - full build complete

now running install, but

line 18006 - second llvm build triggered
ctrl+c pressed

[gyakovlev]

log.zip

[cuviper]

Yes, I do Red Hat packaging, so I'm familiar with how DESTDIR is used.

While it's annoying that the "internal" LLVM install ends up re-copied under DESTDIR, I don't think that's the reason for the complete rebuild. Rather, if you compare the cmake commands on lines 1418 and 18008, the flags are different:

"-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 -m64"
"-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 -m64"

vs.

"-DCMAKE_C_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 -mcpu=native -pipe -frecord-gcc-switches"
"-DCMAKE_CXX_FLAGS=-ffunction-sections -fdata-sections -fPIC -m64 -mcpu=native -pipe -frecord-gcc-switches"

So I guess it's the difference of having your env $(cat config.env) or not, and it's completely natural that this would not be a no-op build.

---

That said, I think we could just use an empty stamp file for non-git builds (from source tarball). This means we would never rebuild LLVM automatically, but it's probably fair to say that folks aren't doing LLVM development in such environments. We could still have a warning to explain that this is happening -- maybe print the path to the stamp file for manually removal to force it anew.

[mati865]

AFAIK multiple rebuilds happen only for cross builds and I'm convinced cross compiling LLVM is screwed up in rustbuild.

Running ./x.py build --host x86_64-pc-windows-gnu --target x86_64-pc-windows-gnu from Linux will fail on building LLVM for Windows (so it's 2nd LLVM build) because somehow LLVM_ON_UNIX is defined.
It gets even weirder when you run build again without cleaning, this time LLVM build will succeed (Rust build will fail because it needs few tweaks).

I'll open new issue for it when I find time to obtain logs, just wanted to make you aware of related bug.

[cuviper]

I think we could just use an empty stamp file for non-git builds (from source tarball).

See #64156 for this.

[psumbera]

I think we could just use an empty stamp file for non-git builds (from source tarball).

See #64156 for this.

I can confirm that this helps on Solaris. Tested with Rust 1.35.0. Thank you!