Build failure: lldb_19

[Ramblurr]

Nixpkgs version

• Unstable (25.05)

Steps to reproduce

git checkout 89e574dd02ba196a29f80ed2eec83f1bb5152c80 # random but recent master commit
nix-build -A lldb 

or for nixos-unstable

git checkout nixos-unstable
nix-build -A lldb 

Can Hydra reproduce this build failure?

Yes, Hydra can reproduce this build failure.

Link to Hydra build job

https://hydra.nixos.org/build/288035026

Relevant log output

ERROR: noBrokenSymlinks: the symlink /nix/store/y02l0yp7lbg19bssf1x1izjyvcppkj7p-lldb-19.1.7-lib/lib/python3.12/site-packages/lldb/lldb-argdumper points to a missing target /nix/store/y02l0yp7lbg19bssf1x1izjyvcppkj7p-lldb-19.1.7-lib/bin/lldb-argdumper
ERROR: noBrokenSymlinks: found 1 dangling symlinks and 0 reflexive symlinks


see below for full build logs

Additional context

lldb-unstable.log

lldb-master.log

System metadata

❯ nix-shell -p nix-info --run "nix-info -m"

• system: "x86_64-linux"
• host os: Linux 6.6.60, NixOS, 24.11 (Vicuna), 24.11pre-git
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.24.10
• nixpkgs: /nix/store/4r8s42c9mwfvgdlxv0izb1cmlzrsb5nz-source

Notify maintainers

---

Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

• I assert that this is a bug and not a support request.
• I assert that this is not a duplicate of an existing issue.
• I assert that I have read the NixOS Code of Conduct and agree to abide by it.

Is this issue important to you?

Add a 👍 reaction to issues you find important.

[Ramblurr]

forgot to ping the eeam; @NixOS/llvm

[RossComputerGuy]

It seems llvm_20 is also affected, this part looks relevant:
https://github.com/llvm/llvm-project/blob/622df0ee9226b90e924538909337d55333d5d2fa/lldb/bindings/python/CMakeLists.txt#L151-L155

[bartman]

I reproduced this on raspberrypi (aarch64) with home-manager setup.

$ nix build 'nixpkgs#lldb'
...
error: builder for '/nix/store/f5jskhf9m14bgwwpdzfy4h2v9jklazs8-lldb-19.1.7.drv' failed with exit code 1;
       last 10 log lines:
       > patching script interpreter paths in /nix/store/n7qjwgnyvkyjg93fq5rvyk5n74rl9g2s-lldb-19.1.7-dev
       > shrinking RPATHs of ELF executables and libraries in /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib
       > shrinking /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib/lib/liblldbIntelFeatures.so.19.1
       > shrinking /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib/lib/liblldb.so.19.1.7
       > checking for references to /tmp/nix-build-lldb-19.1.7.drv-0/ in /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib...
       > patching script interpreter paths in /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib
       > stripping (with command strip and flags -S -p) in  /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib/lib
       > rewriting symlink /nix/store/m2a55zmrdafavxcwqn1grbmivsbs9ag0-lldb-19.1.7/share/vscode/extensions/llvm-org.lldb-dap-0.2.0/bin/lldb-dap to be relative to /nix/store/m2a55zmrdafavxcwqn1grbmivsbs9ag0-lldb-19.1.7
       > ERROR: noBrokenSymlinks: the symlink /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib/lib/python3.12/site-packages/lldb/lldb-argdumper points to a missing target /nix/store/bh01jmr7dlbbclykswsamdhisixq6q3x-lldb-19.1.7-lib/bin/lldb-argdumper
       > ERROR: noBrokenSymlinks: found 1 dangling symlinks and 0 reflexive symlinks
       For full logs, run 'nix log /nix/store/f5jskhf9m14bgwwpdzfy4h2v9jklazs8-lldb-19.1.7.drv'.

full logs are here...

f5jskhf9m14bgwwpdzfy4h2v9jklazs8-lldb-19.1.7.drv.log

[yshui]

i mean adding more checks and be stricter is good, but what i don't get is why can't we make sure it doesn't cause regressions before merging it? this feels very reckless to me... (like, there is tooling to figure out what derivations have stopped building, and we would have to fix them sooner or later anyway. why can't we fix them before merging?)

related: #370750

[RossComputerGuy]

why can't we fix them before merging?

That PR introduced a change which is very large. Building every package which is affected wouldn't be practical. There's a reason why staging exists. It's meant to help iron out issues but not every single issue. According to search.nixos.org, we have over 120,000 packages. Building & checking those would take a long time and a lot of storage.

[ninelore]

I guess it didn't cause enough build failures to be noticed :/

[yshui]

@RossComputerGuy staging exists to catch failures like this, no? we knew this build failed in staging: https://hydra.nixos.org/build/287121996/log/tail

why didn't we just grep the failed builds for "noBrokenSymlinks"? why have hydra and do all these expensive builds if no one even cares?

[RossComputerGuy]

staging exists to catch failures like this, no?

Yes, it would

we knew this build failed in staging

Doesn't help when nobody is constantly watching.

why didn't we just grep the failed builds for "noBrokenSymlinks"?

That would be very computationally heavy and isn't really possible.

[yshui]

why didn't we just grep the failed builds for "noBrokenSymlinks"?

That would be very computationally heavy and isn't really possible.

It wouldn't even take half an hour to search through tens of GBs of logs, there are only 11k failed drvs. and you've already spent much longer than that to build all of them. did anyone even think to try?

Rust builds the entirety of crates.io for changes that might cause backward incompatibility, because they care.

[RossComputerGuy]

I believe the logs are in sqlite, not a plain text file.

[alyssais]

Rust builds the entirety of crates.io for changes that might cause backward incompatibility, because they care.

If you care, you're welcome to help with the staging process (join #staging:nixos.org), and watch for failed builds. It's a balance of how long do we hold back important changes (some of which are security fixes) for the small remaining number of build regressions, so the more people who are working on those regressions, the fewer they will be.

[thefossguy]

A bit offtopic but it would be nice to have a way for Hydra to ping the package maintainer(s) if the package that they're maintaining has a build failure. That way, we can catch it as soon as possible.

[RossComputerGuy]

I believe that's been done before and it resulted in a lot of emails. Also, people would likely end up just ignoring the notification. Anyway, further discussion on this matter should move to Matrix.

[Ramblurr]

Conversation aside (though I find it interesting and relevant fwiw), does anyone know how to fix it? lldb is a dependency of jetbrains ides, so they are broken atm.

[gepbird]

Conversation aside (though I find it interesting and relevant fwiw), does anyone know how to fix it? lldb is a dependency of jetbrains ides, so they are broken atm.

A workaround for it is to disable the broken symlink checking behavior:

  nixpkgs.overlays = [
    (final: prev: {
      lldb = prev.lldb.overrideAttrs {
       dontCheckForBrokenSymlinks = true;
      };
    })
  ];

(I only tested it for other packages, and lldb is taking a while to build locally)

[yshui]

you're welcome to help with the staging process

i'd love to help, but i have too many commitments right now and if i take on more i'll just end up doing none of them. but besides manpower, i (looking from outside) feel like there might be a tooling problem? if it was easy to query "what did noBrokenSymlinks break", then someone probably would have done that.

A workaround for it is to disable the broken symlink checking behavior:

if you do this, make sure you override the right one. for me it was lldb_14 used by the vscode lldb extension :/

I believe that's been done before and it resulted in a lot of emails

did it send an email each time a drv was rebuilt? then i can imagine it being super annoying. maybe it's possible to do something smarter? or maybe manually? e.g. when devs feel staging is mature enough, they can trigger emails to be sent for remaining broken drvs.

[paparodeo]

removing the lib output seems like it would be the easiest thing to do given it already doesn't exist on darwin and both outputs (out, lib) are needed for for lldb to work anyway.

diff --git a/pkgs/development/compilers/llvm/common/lldb.nix b/pkgs/development/compilers/llvm/common/lldb.nix
index f4a131c47620..820bd5f21231 100644
--- a/pkgs/development/compilers/llvm/common/lldb.nix
+++ b/pkgs/development/compilers/llvm/common/lldb.nix
@@ -56,7 +56,7 @@ stdenv.mkDerivation (rec {
 
   # LLDB expects to find the path to `bin` relative to `lib` on Darwin. It can’t be patched with the location of
   # the `lib` output because that would create a cycle between it and the `out` output.
-  outputs = [ "out" "dev" ] ++ lib.optionals (!stdenv.hostPlatform.isDarwin) [ "lib" ];
+  outputs = [ "out" "dev" ];
 
   sourceRoot = lib.optional (lib.versionAtLeast release_version "13") "${src.name}/${pname}";
 

[RossComputerGuy]

I think the fix which would make sense is to patch this: #380196 (comment)

[paparodeo]

I think the fix which would make sense is to patch this: #380196 (comment)

disagree given the added maintenance burden for pretty much zero benefit, not to mention that it doesn't work given it creates a cycle which breaks the build.

[bartman]

here is my attempt at cleaning up the dangling symlink...
#380669
(apparently not the right thing to do... ignore)

[szaffarano]

Conversation aside (though I find it interesting and relevant fwiw), does anyone know how to fix it? lldb is a dependency of jetbrains ides, so they are broken atm.

A workaround for it is to disable the broken symlink checking behavior:

nixpkgs.overlays = [
(final: prev: {
lldb = prev.lldb.overrideAttrs {
dontCheckForBrokenSymlinks = true;
};
})
];

(I only tested it for other packages, and lldb is taking a while to build locally)

☝ this workaround worked for me, thanks a lot @gepbird