linker-plugin-lto stopped working in Rust 1.45.0 #74657
Comments
jonas-schievink
added
A-linkage
rustbot
added
the
I-prioritize
|
In https://bugzilla.mozilla.org/show_bug.cgi?id=1654465, Mozilla is switching from setting |
Add a patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1654465 Switch from ld.bfd to ld.lld to work around rust-lang/rust#74657 git-svn-id: file:///srv/repos/svn-packages/svn@392595 eb2447ed-0c53-47e4-bac8-5bc4a241df78
Add a patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1654465 Switch from ld.bfd to ld.lld to work around rust-lang/rust#74657 git-svn-id: file:///srv/repos/svn-packages/svn@392595 eb2447ed-0c53-47e4-bac8-5bc4a241df78
|
The issue is also reproducible on Ubuntu 20.04 with rust 1.45.0 installed using rustup.rs. Bisecting points to 1357af3 which sounds relevant. For the bisect, I used |
|
@rustbot ping cleanup To see if we can minimize this issue. |
rustbot
added
the
ICEBreaker-Cleanup-Crew
|
Hey Cleanup Crew ICE-breakers! This bug has been identified as a good cc @AminArria @camelid @chrissimpkins @contrun @DutchGhost @elshize @ethanboxx @h-michael @HallerPatrick @hdhoang @hellow554 @imtsuki @kanru @KarlK90 @LeSeulArtichaut @MAdrianMattocks @matheus-consoli @mental32 @nmccarty @Noah-Kennedy @pard68 @PeytonT @pierreN @Redblueflame @RobbieClarken @RobertoSnap @robjtede @SarthakSingh31 @senden9 @shekohex @sinato @spastorino @turboladen @woshilapin @yerke |
|
Assigning |
spastorino
added
P-high
LeSeulArtichaut
added
the
E-needs-mcve
|
|
From Zulip:
To clarify, the reason I believe this isn't hitting anyone else but Arch Linux yet is that we're the only ones building Firefox with cross-language LTO and Rust 1.45. Mozilla's official builds use an older Rust version, which isn't affected. Developer builds don't do LTO or use LLD if it's available, which isn't affected.
Not this Arch Linux applies two fixes to its Firefox build:
|
Add a patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1654465 Switch from ld.bfd to ld.lld to work around rust-lang/rust#74657 git-svn-id: file:///srv/repos/svn-community/svn@665789 9fca08f4-af9d-4005-b8df-a31f2cc04f65
Add a patch for https://bugzilla.mozilla.org/show_bug.cgi?id=1654465 Switch from ld.bfd to ld.lld to work around rust-lang/rust#74657 git-svn-id: file:///srv/repos/svn-community/svn@665789 9fca08f4-af9d-4005-b8df-a31f2cc04f65
|
Actually, older versions of rust are affected if you switch -Clto for -Clto=thin, but not on all platforms. |
|
Is there perhaps a small/simple docker container to reproduce this? I unfortunately can't reproduce this locally at this time. |
|
This Dockerfile reproduces the issue with |
|
It looks like I've configured my system for Is this an issue with |
|
In the Firefox case, rustc is not invoking ld: it generates a staticlib that then is linked into libxul. |
|
At this point of my investigation, I'm not sure if it's a BFD ld bug or a LLVM gold plugin bug yet, but I have a smaller reproducer that is independent of rust (but emulates the problem): https://gist.github.com/glandium/98f8d5dd2e075860a731ed374dd284a1 Both The reason why it doesn't fail with gold or lld is that they don't even try to use the bitcode, they just use the code in The same failure happens on x86 for things like One way this could be worked around in rust (because even if the bug, wherever it is, is fixed, it will take a while for the fix to make its way everywhere) would be to avoid embedding bitcode for the object files containing these special symbols. Essentially, all of libcompiler_builtins... And considering only BFD ld uses the bitcode anyways... |
|
Unfortunately, there's no way that I can see to opt-out of Line 289 in dd7fc54 |
|
I'm leaning towards a problem on LLVM's end, so I filed https://bugs.llvm.org/show_bug.cgi?id=47872 |
jyn514
added
the
A-LLVM
Are you sure that BFD ld is using the bitcode? How does it know that there is bitcode embeded in probestack.o? |
Answering my own question, I can get it to work if I strip out the the .llvmbc section: The other question I have is if gold and lld ignore the embedded bitcode, then does that mean they aren't actually doing LTO? |
|
I spent some time debugging this and what I found is that bfd passes libprobestack.a to the LLVMgold plugin, but the plugin refuses to claim it, because it does not support reading archives. Therefore, the bitcode from libprobestack.o is never considered during LTO. Then after LTO is done, bfd passes the libprobestack.o member of libprobestack.a to the plugin. The pugin sees the embedded bitcode in libprobestack.o and claims this file. However, at this point it is too late for bfd to add new IR symbols to the link, so the file gets dropped. My guess is that also because the plugin claimed the file, then bfd does not include any of the ELF symbols from the file in the link. I confirmed this by telling the plugin not to claim the libprobestack.o member and this makes the example link. The reason adding probestack() to main.c works, is because when you do this, bfd passes the libprobestack.o member of lbprobestack.a to the plugin before LTO is run, so the bitcode and symbols from libprobestack.o participate in LTO. gold on the other hand does not even attempt to pass the libprobestack.a archive or its probestack.o member to the plugin, and I think it ends up getting the ELF symbols from those files. It's possible there is a bug in bfd here, I'm not sure, but it looks like we could fix the LLVMgold plugin in two different ways:
|
Personally I would say that this is an LLVMgold plugin bug, since it is that plugin's responsibility to handle LLVM bitcode files. But I could be persuaded otherwise. If you do think that the BFD library should be doing something differently, please could you file a bug report here:
I think that this would be bad, because the plugin will not know which members of the archive are going to be needed to complete the link. Unless it is possible to compile each of the members in isolation and then build a new archive containing these processed object files and return that to the linker.
This sounds like a good idea. Even if the bug really is with the BFD library, I think that the LLVM plugin would benefit from this check. |
For the record, that bug is now llvm/llvm-project#47216 |
|
@alexcrichton Is the intention with 1357af3 that linkers will always ignore the .llvmbc section? If so, then whatever mechanism was added to cause linkers to ignore the section is not working with bfd and the gold plugin. I also wonder if this is fixed by #90326. |
|
Yes that was the original intention. We could probably rename the section to something like |
|
@alexcrichton I think renaming the section to .rustc_llvmbc is going to be the safest solution for rust if you want to guarantee that the section gets ignored. The .llvmbc has special meaning in LLVM, so even if we were to update the gold plugin to ignore it (which may be possible), there could be other problems that pop up in the future as new features are added to LLVM. Renaming the section is also going to be the only way to ensure this bug is fixed for LLVM <=13, since those versions are no longer getting fixes upstream. |
|
IIRC we also rely on LLVM-internal functions to find the bytecode within an object, and I think those functions require the section to be named |
|
Visiting for P-high review From what @alexcrichton has written above, it sound like there are clear ideas for a straight-forward fix here; and the issue is just figuring out the all the internal plumbing for making this change. I'm going to self-assign with the hopes of using it as the basis for some instructional videos on how to hack on @rustbot claim |
|
Actually this might now be fixed upstream in LLVM, based on https://reviews.llvm.org/rG4b1e3d19370694dd2b2c04a5945f3f9e43917456 (committed back on 2022-07-14, which is in LLVM 15, which we started using in x86 CI dist builds back in #101527) So maybe this is fixed now? Seems worth checking. (But even if it is fixed, the earlier comment implies that there might still be value in renaming the relevant section here to |
|
@heftig can you confirm whether or not this has been resolved for you by a more recent Rust release that is using LLVM 15 (and thus has the fix here)? (I spent a little while trying do such confirmation on my own, but I'm not well versed in how to adjust Rust's LLVM build settings to get access to the LLVMgold.so plugin.) |
|
I currently get another error trying to reproduce this: |
|
You are using a gold plugin from LLVM 14 to read bitcode produced by LLVM 15, which is not possible. |
|
This has been resolved. |
After upgrading from Rust 1.44.1 to Rust 1.45.0, Firefox's LTO build failed to link a crate build script with
undefined reference to '__rust_probestack'.The system is Arch Linux, which has binutils 2.34, GCC 10.1.0, LLVM 10.0.0, Clang 10.0.0.
Originally posted at https://bugzilla.mozilla.org/show_bug.cgi?id=1640982#c30:
I can replicate the error with rustup and the rusqlite crate (https://github.com/rusqlite/rusqlite) like this:
This works with 1.44.1:
Using
clang -fltoas the linker instead of GCC with explicitLLVMgold.sobehaves analogously:Using LLD or gold instead of the BFD linker also makes it work:
The text was updated successfully, but these errors were encountered: