Build failure: gitstatus on x86_64-darwin

[pitkling]

Steps To Reproduce

Steps to reproduce the behavior:

1. build gitstatus on x86_64-darwin

Build log

Build Log

Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/pn5xs822bapfv625apn8yddylclv79qh-source
source root is source
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
no configure script, doing nothing
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
build flags: SHELL=/nix/store/fhngvmgagwwxhak8nzrx5vqbg4g4qiac-bash-5.2p37/bin/bash
mkdir -p -- obj
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/tag_db.o src/tag_db.cc >obj/tag_db.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/tag_db.o src/tag_db.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/arena.o src/arena.cc >obj/arena.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/arena.o src/arena.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/strings.o src/strings.cc >obj/strings.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/strings.o src/strings.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/gitstatus.o src/gitstatus.cc >obj/gitstatus.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/gitstatus.o src/gitstatus.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/repo_cache.o src/repo_cache.cc >obj/repo_cache.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/repo_cache.o src/repo_cache.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/repo.o src/repo.cc >obj/repo.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/repo.o src/repo.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/thread_pool.o src/thread_pool.cc >obj/thread_pool.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/thread_pool.o src/thread_pool.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/timer.o src/timer.cc >obj/timer.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/timer.o src/timer.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/request.o src/request.cc >obj/request.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/request.o src/request.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/response.o src/response.cc >obj/response.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/response.o src/response.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/check_dir_mtime.o src/check_dir_mtime.cc >obj/check_dir_mtime.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/check_dir_mtime.o src/check_dir_mtime.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/git.o src/git.cc >obj/git.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/git.o src/git.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/logging.o src/logging.cc >obj/logging.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/logging.o src/logging.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/options.o src/options.cc >obj/options.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/options.o src/options.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/index.o src/index.cc >obj/index.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/index.o src/index.cc
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -MM -MT obj/dir.o src/dir.cc >obj/dir.dep
clang++ -std=c++14 -funsigned-char -O3 -DNDEBUG -DGITSTATUS_VERSION=v1.5.5 -Wall  -Wall -c -o obj/dir.o src/dir.cc
clang++  obj/tag_db.o  obj/arena.o  obj/strings.o  obj/gitstatus.o  obj/repo_cache.o  obj/repo.o  obj/thread_pool.o  obj/timer.o  obj/request.o  obj/response.o  obj/check_dir_mtime.o  obj/git.o  obj/logging.o  obj/options.o  obj/index.o  obj/dir.o -pthread  -lgit2  -o usrbin/gitstatusd
ld: library not found for -lgit2
clang-16: �[0;1;31merror: �[0m�[1mlinker command failed with exit code 1 (use -v to see invocation)�[0m
make: *** [Makefile:25: usrbin/gitstatusd] Error 1

Additional context

I stumbled upon this when updating to 24.11 and building zsh-powerlevel10k (which depends on gitstatus). The builds work fine on aarch64_darwin and x86_64-linux. I looked a little bit into it and it seems that NIX_LDFLAGS is not populated with the path to libgit2 under x86_64-darwin, causing the linker error above.

I guess this is related to the recent changes in the handling of Apple's SDKs. As far as I know the bintools-wrapper is responsible for populating NIX_LDFLAGS. Since there were a few changes to the bintools-wrapper by @emilazy related to the sdk update (see "Known Issues" under The Darwin SDKs have been updated), I'm pinging them here, in case that is related. Hope that is ok.

Metadata

• system: "x86_64-darwin"
• host os: Darwin 22.6.0, macOS 10.16
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.18.8
• channels(peter): ""
• channels(root): ""
• nixpkgs: /nix/store/9rlx9vnfg79i0y9nqfa1578jml31d5vr-source

Notify maintainers

@mmlb @Hexa @SuperSandro2000

---

Note for maintainers: Please tag this issue in your PR.

---

Add a 👍 reaction to issues you find important.

[SuperSandro2000]

Can you build the normal libgit2? We are just providing it with a few overrides to the cmake flags. Other than that I cannot help you much because I don't own a Mac.

[pitkling]

Building libgit2 is actually not the problem. I checked and the implicit build of libgit2 with the overrides of the cmake flags succeeds and gives the desired libgit2.a in the nix store.

To me it really seems that even though buildInputs of gitstatus includes the (implicit build of) libgit2, the bintools-wrapper for some reason does not add the library's location to NIX_LDFLAGS under x86_64-darwin.

I can try to investigate this further, just waiting a bit to see whether @emilazy has an idea and can point me into the right direction.

[emilazy]

I have very little idea why it’d be failing only on x86_64-darwin, I’m afraid. I assume we have to use that libgit2 fork and can’t use upstream? That seems problematic in its own right, as it hasn’t been updated in a year and there have been libgit2 CVEs since.

[DaGenix]

This patch fixes things for me:

diff --git a/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix b/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix
index 9881bd480406..f2015af2069d 100644
--- a/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix
+++ b/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix
@@ -3,7 +3,7 @@
 libgit2.overrideAttrs (oldAttrs: {
   cmakeFlags = oldAttrs.cmakeFlags ++ [
     "-DBUILD_CLAR=OFF"
-    "-DBUILD_SHARED_LIBS=OFF"
+    "-DBUILD_SHARED_LIBS=ON"
     "-DREGEX_BACKEND=builtin"
     "-DUSE_BUNDLED_ZLIB=ON"
     "-DUSE_GSSAPI=OFF"

[DaGenix]

The difference is that with this change, the forked libgit2 produces .dylib outputs instead of a .a. And, I dunno why that fixes the issue on x86_64-darwin, but, it seems to.

[pitkling]

@emilazy Thanks anyway for taking a look! And I think yes, using upstream libgit2 seems difficult, as the fork is described as including quite extensive changes. There is also an update request to libgit2 1.8.0 (which would at least include the CVEs). I asked the fork owner in that issue whether there's any chance to see an update to a version containing the CVEs. He just replied that currently that's unlikely, also since gitstatus uses only a small part of libgit2 that's not related to the CVEs.

@DaGenix Thanks! I cannot test this right now myself but will do so later. Odd that x86_64-darwin doesn't catch the static lib while other platforms seem to do so. I did a quick search about nixpkgs, static libraries and NIX_LDFLAGS. There is a very old, unanswered question on discourse about a static library not being picked up on x86_64-darwin. But then, picking up the static library seemingly worked on x86_64-darwin in 24.05, so that's probably unrelated.

Given that shared libs were explicitly disabled, I assume there must be some reason for not using them?

[DaGenix]

@pitkling Yah, that's the part I have no explanation for. Also, it was building in Hydra until about 3-4 weeks ago, which makes me think that it must have been something else that changed that caused the new behavior. I considered opening up a PR with this change, but, as you say, it was made a static library for a reason and also I suspect this is just a symptom of a more complex behavior.

[pitkling]

The following change in pkgs/by-name/gi/gitstatus /romkatv_libgit2.nix (see also this patch) preserves the static linking and seems to work for me on both x86_64-darwin and aarch64-darwin:

(libgit2.override { staticBuild = true; }).overrideAttrs (oldAttrs: {
  cmakeFlags = oldAttrs.cmakeFlags ++ [
    "-DBUILD_CLAR=OFF"
    "-DREGEX_BACKEND=builtin"
    "-DUSE_BUNDLED_ZLIB=ON"
    "-DUSE_GSSAPI=OFF"
    "-DUSE_HTTPS=OFF"
    "-DUSE_HTTP_PARSER=builtin" # overwritten from libgit2
    "-DUSE_NTLMCLIENT=OFF"
    "-DUSE_SSH=OFF"
    "-DZERO_NSEC=ON"
  ];

Again, no idea why this affects only x86_64-darwin… Seems like the staticBuild argument is affecting other parts hidden inside mkDerivation.

[trofi]

Was it already bisected by chance by anyone? Otherwise I'll start a slow bisect.

[DaGenix]

I didn't

[trofi]

Bisect landed on a061b02 is the first bad commit Revert "python{27,39,310,311,312,313,314}: use a bootstrap SDK on Darwin".

No idea how/why it affects the presence on static builds for libgit2. Perhaps the bootstrapped sdk has very different properties compared to the default one. /cc @emilazy

Reverting it against master fixes the gitstatus build. The whole revert:

$ git diff
$ diff --git a/pkgs/development/interpreters/python/cpython/default.nix b/pkgs/development/interpreters/python/cpython/default.nix
index 20c7c0c145ef..a7b34e3329d1 100644
--- a/pkgs/development/interpreters/python/cpython/default.nix
+++ b/pkgs/development/interpreters/python/cpython/default.nix
@@ -180,7 +180,7 @@ let
     darwin.apple_sdk.frameworks.Cocoa
   ] ++ optionals stdenv.hostPlatform.isDarwin [
     # Work around for ld64 crashes on x86_64-darwin. Remove once 11.0 becomes the default.
-    apple-sdk_11
+    (apple-sdk_11.override { enableBootstrap = true; })
   ] ++ optionals stdenv.hostPlatform.isMinGW [
     windows.dlfcn
     windows.mingw_w64_pthreads

[emilazy]

Hmm, I introduced that in c455166, the same day I reverted it (there was a couple rounds of botched fixes for an issue). Are you confident in that bisect result? It might have been one of the other related changes that broke it.

[trofi]

Ah, maybe it was the first one. I'll try again.

[trofi]

Wait, I did test the revert against master. It's unreverted for is certainly present in master.

[emilazy]

Ah, sorry, I mean that your bisected commit a061b02 is a revert of my commit c455166 from the same day. So if this was working before the staging cycle those were in, then you probably want to skip that day’s churn and look earlier. (But it’s also possible that the changes that were related to that one from around the same time might be the cause.)

It is in any event very strange to me that adding the bootstrap SDK back fixes this. I cannot fathom why.

[trofi]

I compared the build logs of "good" and "bad" libgit2 fork. Looks like one of the closures is missing ... grep:

--- /dev/fd/63  2024-11-25 16:04:03.792107343 +0000
+++ /dev/fd/62  2024-11-25 16:04:03.792107343 +0000
@@ -1,12 +1,10 @@
+/<<NIX>>/cmake-3.30.5/nix-support/setup-hook: line 77: grep: command not found
...
 stripping (with command strip and flags -S) in  /<<NIX>>/libgit2-1.8.4-dev/lib
 checking for references to /private/tmp/nix-build-libgit2-1.8.4.drv-0/ in /<<NIX>>/libgit2-1.8.4...
 patching script interpreter paths in /<<NIX>>/libgit2-1.8.4
+/<<NIX>>/cmake-3.30.5/nix-support/setup-hook: line 189: grep: command not found
+/<<NIX>>/multiple-outputs.sh: line 201: grep: command not found

Looks like darwin's stdenv lacks grep. That sounds bad.

The following workaround is enough to fix gitstatus build for me:

--- a/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix
+++ b/pkgs/by-name/gi/gitstatus/romkatv_libgit2.nix
@@ -1,4 +1,4 @@
-{ fetchFromGitHub, libgit2, ... }:
+{ fetchFromGitHub, libgit2, gnugrep, ... }:

 libgit2.overrideAttrs (oldAttrs: {
   cmakeFlags = oldAttrs.cmakeFlags ++ [
@@ -26,4 +26,6 @@ libgit2.overrideAttrs (oldAttrs: {
   doCheck = false;

   patches = [ ];
+
+  nativeBuildInputs = oldAttrs.nativeBuildInputs ++ [ gnugrep ];
 })

[trofi]

Proposed gitstatus gnugrep addition hack as:

• gitstatus: add grep to nativeBuildInputs for darwin #359036

[emilazy]

Uh, we had a Darwin Hydra builder recently that was missing grep and it turned out to be store corruption. I’m wondering if something related is going on here.

[paparodeo]

grep is in the stdenv. https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/common-path.nix#L6 so there is some other issue going on causing it to fail to build but report success. if you have access to the environment which is failing can you look at the path? it probably has a gnugrep in there with an empty bin directory. getting a log from that build would be helpful.

[trofi]

Ah, poking at the environment is a good point. I tried to rebuild the libgit2 locally and it built fine (and failed reproducibility check). It indeed looks like a problem of a poisoned cached build (like a GC in the middle of the build, or an occasionally failing build).

[trofi]

Let's try to unpoison hydra instead by introducing minor rebuild to see if it was this derivation corruption, or something deeper in the build chain:

• gitstatus: trigger rebuild (avoid poisoned darwin entry) #359133

[paparodeo]

I didn't realize nix log /nix/store/blbb8bhxqhpaif73xqnw0n0msf3k5fhl-libgit2-1.8.4 would provide the hydra log and that this is on current master.

diffing the outputs from the hydra and and local-built, besides the libgit2.a differing, the propagated-build-inputs on hydra is missing the good parts that point to libgit2.

--- result-dev/nix-support/propagated-build-inputs	1969-12-31 16:00:01.000000000 -0800
+++ local-built/result-dev/nix-support/propagated-build-inputs	1969-12-31 16:00:01.000000000 -0800
@@ -1 +1 @@
-/nix/store/5b7ddkjx3bp5yv3rf172dqp6f2957l2g-libiconv-107-dev 
\ No newline at end of file
+/nix/store/5b7ddkjx3bp5yv3rf172dqp6f2957l2g-libiconv-107-dev  /nix/store/sbrs050xq1vjxcbcji66rfl81njwdmrv-libgit2-1.8.4-lib /nix/store/hzzb6p3nxrkypcxkkha8a3qx4l3fa33j-libgit2-1.8.4
\ No newline at end of file

[paparodeo]

nix log /nix/store/blbb8bhxqhpaif73xqnw0n0msf3k5fhl-libgit2-1.8.4

/nix/store/6paqqjczjfv8z2i6qr7gdfq21n0rbglr-cmake-3.30.5/nix-support/setup-hook: line 189: grep: command not found
/nix/store/jivxp510zxakaaic7qkrb7v1dd2rdbw9-multiple-outputs.sh: line 201: grep: command not found

both disable failures. the first uses || true and the second disables pipefail which allows the build to succeed with a broken propagated-build-inputs

nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh

Lines 195 to 203 in b718c27

	# Default value: propagate binaries, includes and libraries
	if [ -z "${propagatedBuildOutputs+1}" ]; then
	local po_dirty="$outputBin $outputInclude $outputLib"
	set +o pipefail
	propagatedBuildOutputs=`echo "$po_dirty" \
	| tr -s ' ' '\n' | grep -v -F "$propagaterOutput" \
	| sort -u | tr '\n' ' ' `
	set -o pipefail
	fi

nixpkgs/pkgs/by-name/cm/cmake/check-pc-files-hook.sh

Lines 1 to 16 in b718c27

	cmakePcfileCheckPhase() {
	while IFS= read -rd $'\0' file; do
	grepout=$(grep --line-number '}//nix/store' "$file" || true)
	if [ -n "$grepout" ]; then
	{
	echo "Broken paths found in a .pc file! $file"
	echo "The following lines have issues (specifically '//' in paths)."
	echo "$grepout"
	echo "It is very likely that paths are being joined improperly."
	echo 'ex: "${prefix}/@CMAKE_INSTALL_LIBDIR@" should be "@CMAKE_INSTALL_FULL_LIBDIR@"'
	echo "Please see https://github.com/NixOS/nixpkgs/issues/144170 for more details."
	exit 1
	} 1>&2
	fi
	done < <(find "${!outputDev}" -iname "*.pc" -print0)
	}

[paparodeo]

looking at the set -x output of libgit2 it seems that grep was called 5 times before the first failure so at one point it was in the path -- so I guess this points to a GC during the build process.

[paparodeo]

once staging-next is merged #360437 the issue should be fixed
https://hydra.nixos.org/build/280376521

[pitkling]

It is actually fixed (at least using the nixpkgs-24.11-darwin branch), so I'm closing this. Thanks everybody!

[vcunat]

The current staging-next is very close to what now landed on 24.11.