impossible to link (ld / clang) liboping on macOS Big Sur 11 Beta

[sycured]

Describe the bug
Compilation fail

To Reproduce
Steps to reproduce the behavior:

1. add this line into ~/.config/nixpkgs/config.nix otherwise it's impossible to install on macOS:

( pkgs.liboping.overrideDerivation (old: {NIX_CFLAGS_COMPILE = "";} ) )

2. nix-env -i all

Expected behavior
Don't fail and arrive to do the linking (ld)

Output

building '/nix/store/0psa9x9abc3yhv4r2pnw44q0ywi15v4y-liboping-1.10.0.drv'...
unpacking sources
unpacking source archive /nix/store/x1f1kszhpxb4x4c8j9dqag614kyx56d6-liboping-1.10.0.tar.bz2
source root is liboping-1.10.0
setting SOURCE_DATE_EPOCH to timestamp 1494487978 of file liboping-1.10.0/src/mans/ping_iterator_get_info.3
patching sources
configuring
fixing libtool script ./ltmain.sh
configure flags: --disable-static --disable-dependency-tracking --prefix=/nix/store/s45i6bciaw48mv9s0xfwhh3m7vpiqy8s-liboping-1.10.0
checking for a BSD-compatible install... /nix/store/j0xbl8447zrk5f97d6w55d2l8qaldjxy-coreutils-8.31/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /nix/store/j0xbl8447zrk5f97d6w55d2l8qaldjxy-coreutils-8.31/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... clang
checking whether the C compiler works... no
configure: error: in `/private/var/folders/g5/ljjjnl3x4fq4x9tg69wy3lqh0000gn/T/nix-build-liboping-1.10.0.drv-0/liboping-1.10.0':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/0psa9x9abc3yhv4r2pnw44q0ywi15v4y-liboping-1.10.0.drv' failed with exit code 77
cannot build derivation '/nix/store/qbsyr9i6wrv43rnfjdw9v2nf9kgav62f-all.drv': 1 dependencies couldn't be built
error: build of '/nix/store/qbsyr9i6wrv43rnfjdw9v2nf9kgav62f-all.drv' failed

Notify maintainers

Contributors from https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/liboping/default.nix:

• @fpletz
• @bjornfor
• @volth
• @OmnipotentEntity
• @globin
• @Ericson2314

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-darwin"
• host os: Darwin 20.0.0, macOS 10.16
• multi-user?: no
• sandbox: no
• version: nix-env (Nix) 2.3.6
• channels(sycured): "nixpkgs-20.09pre231837.2cd2e7267e5"
• nixpkgs: /Users/sycured/.nix-defexpr/channels/nixpkgs

host os detected isn't real because it's macOS 11.0 Beta, not 10.16 :

[mroi]

This is a general issue on Big Sur and not specific to liboping. For example, I can also reproduce this using:

nix-build --check -A hello '<nixpkgs>'

The issue comes down to the new dyld cache and the subsequent removal of libraries from their usual filesystem locations. I am currently experimenting with updating the linker to address this issue. I will open a PR and mention it here, if I find a solution.

[mroi]

I updated the linker to version 530 from the cctools port used by Nix. This is the latest version Apple currently provides as open source. However, I cannot get simple C code to link with it. This is the error message:

ld: file not found: /usr/lib/system/libcache.dylib for architecture x86_64

The same object files link successfully with the linker shipping with Xcode 11.5 (ld version 556.6). It’s possible we have to wait until Apple releases this linker version’s source.

[mroi]

I created an overlay, with which I can successfully build the hello package on a Big Sur machine. It recruits the platform’s linker to do the actual work, so it is an ugly hack, but it works on first tests:

(self: super: {
	darwin = super.darwin // {
		cctools = super.darwin.cctools.overrideAttrs (attrs: {
			# workaround by using platform linker
			postInstall = ''
				rm $out/bin/ld
				cat <<- "EOF" > $out/bin/ld
					#!/bin/sh
					exec /usr/bin/ld -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk "$@"
				EOF
				chmod a+x $out/bin/ld
			'';
		});
	};
})

[sycured]

Not working

checking whether the C compiler works... no
configure: error: in `/private/var/folders/g5/ljjjnl3x4fq4x9tg69wy3lqh0000gn/T/nix-build-cctools-port.drv-0/source/cctools':
configure: error: C compiler cannot create executables
See `config.log' for more details
builder for '/nix/store/3j35kh281l86jrfwaahymy64f2svn6p5-cctools-port.drv' failed with exit code 77
cannot build derivation '/nix/store/nash3pyyz99vgqq5mrxk1d20sqcm1si8-Libsystem-osx-10.12.6.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/fnlcxyalmqgcnbsry94l74nc2v89fhiy-binutils-wrapper-2.31.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8ffwdnrilnn3zsmcrqzn01w09hh6w3wb-bootstrap-stage2-clang-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/lgz47r57d8k3a6m1mv6cbpds9g2svanx-bootstrap-stage2-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/rn7jxh4vx80by5l34r80zp2g22s6l624-bootstrap-stage2-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vzxd930xxgy44knvcfgkhnr5p82v1vn4-bootstrap-stage3-clang-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/apb4q462djzfj93pgjp66zksbfbfax64-bootstrap-stage3-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/q6fzrm5wid7ivy9173vynnflmpia1g9z-bootstrap-stage3-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ixjkpmkwhxszb11y5dszsqkniz8kgahf-bootstrap-stage4-clang-wrapper-.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/qs2vqg24qma6r5g1ms077jr5h3blg1jk-cctools-binutils-darwin-wrapper-927.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/mf4z96pkyhj3i1rgxcjmzfllz305m6ja-clang-wrapper-5.0.2.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/sfxm7hvj3vzll75y1cbkdvv2l0x444jp-clang-wrapper-6.0.1.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/vhynnghj1qmany8mgh21a9cdzdgq9ymm-clang-wrapper-7.1.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/0mhv73mm2k6vq7xgy8a4gz8cl9ixxw5q-cmake-3.17.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/4yag260d3zfp5d6y5y8l5kkwz9rvgxaa-cmake-3.17.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/8fk1jnw24mzqc004c5lj5qli0fmjqbis-gcc-9.3.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/763nvn0vji4mljg2dyx87knh4pcji81f-perl-5.30.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/g72k2p18r5f8pb5ka7m37nc8xacfn5nf-perl-5.30.3.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/b2dhb60k122z3z8xkz8ja4mqm1jmjnmr-python-2.7.18.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/b1z965fh4lfpb7hpmi18mk4831ji34qq-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/ixxvnwdrpgi7vf0pk1fiq6lfxq1aa410-stdenv-darwin.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/17j6hyfrfl6glmri6zb0lhgr5mf2y4l2-all.drv': 1 dependencies couldn't be built
error: build of '/nix/store/17j6hyfrfl6glmri6zb0lhgr5mf2y4l2-all.drv' failed

[mroi]

Yes, I forgot to explain that properly: You cannot bootstrap my workaround on Big Sur, since it tries to build my hacked cctools package with the current cctools linker, which will fail in the way you show. You need to build the stdenv on a pre-Big-Sur machine and then copy the closure over to the Big Sur machine.

[sycured]

Thank you, I'm trying with a VM (Catalina)

I received this message from Apple Feedback because I opened an issue before opening the issue in this github:

The macOS Big Sur has the following change that’s documented in the release notes -> https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11-beta-release-notes:

New in macOS Big Sur 11 beta, the system ships with a built-in dynamic linker cache of all system-provided libraries. As part of this change, copies of dynamic libraries are no longer present on the filesystem. Code that attempts to check for dynamic library presence by looking for a file at a path or enumerating a directory will fail. Instead, check for library presence by attempting to dlopen() the path, which will correctly check for the library in the cache.

This means that we no longer ship individual dylib files on disk. You should try passing in the -isysroot <PATH_TO_MACOS_SDK> flag to your clang link invocation.

[sycured]

I tried your overlay on Catalina and it's not working in my case

At each test, I obtain this error… @mroi have you an idea?

[mroi]

Could you give me a nix-build command to reproduce?

[sycured]

@mroi I tried nix-env -i ansible
But it's the same with nix-build -A ansible '<nixpkgs>'

[mroi]

I can reproduce the problem here. The newer linker version appears to be more picky regarding its options than the old one.
I guess this just shows how flaky this workaround is. :(

I think we just have to wait for Apple to update their linker’s source release and then integrate it step by step. Unless someone else has a better solution?

[sycured]

No idea excepted looking at https://opensource.apple.com/source/cctools/ and waiting the release of new version

[domenkozar]

@thefloweringash this is fixes is staging right?

[thefloweringash]

@thefloweringash this is fixes is staging right?

Yeah. It builds and links for me on my Big Sur machine.

[nasyxx]

@thefloweringash this is fixes is staging right?

Yeah. It builds and links for me on my Big Sur machine.

Hi, I just updated to big sur and met this problem. How can I use the new tdb stdenv?
Simply change nixpkgs to staging branch?

[tscholak]

Same here, is it possible to set the nix channel to staging-next somehow?

[malob]

@nasyxx, @tscholak, I was able to get things working on Big Sur by targeting the staging-next branch. I don't use nix-channel to manage my package sets but instead manage my them with niv by symlinking in a custom folder to replace ~/.nix-defexpr (more details in the first bullet here).

But nothing that fancy is required, you should be able to do something like the following:

nix-channel --add https://github.com/NixOS/nixpkgs/archive/staging-next.tar.gz nixpkgs
nix-channel update

Do note though that building from staging-next will cause a very large number of rebuilds (since there will be fewer cache hits), and will likely take a long time.

[nasyxx]

@malob, It seems that the staging-next branch does not work.

ld: file not found: /usr/lib/system/libcache.dylib
builder for '/Users/Nasy/.nix/store/kwb79x0vnax2s9a6liggizdgn7kiwhpp-bootstrap-tools.drv' failed with exit code 1

[malob]

@nasyxx, oh no!

One thing to note is that I built my config on Catalina using staging-next before updating to Big Sur. Seems likely that Nix can't bootstrap itself on Big Sur yet. I don't have a deep understanding of the changes to support Big Sur, so hopefully others who do can chime in.

[nasyxx]

I see!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-there-a-plan-that-nix-can-bootstrap-itself-on-macos-big-sur/10043/1

[domenkozar]

@nasyxx once binary cache is available you can just use that.

[nasyxx]

@domenkozar Thanks! But which binary is it? Will it be usable if I build nix from source?

[fiksn]

This is quite frustrating. Is there any easy way to boot to Catalina from USB key or sth? So I would be able to build everything from staging-next and go back to Big Sur. Or if somehow the whole darwin "closure" from staging-next was in some cache. Is anyone willing to share their stuff? Or what did you mean @domenkozar with "once binary cache is available". When will that be - at least approximately?

[yihuang]

it seems works now, after updated to nixpkgs master.

[nasyxx]

it seems works now, after updated to nixpkgs master.

No, it does not work. Stop on boostrap-tools. The bin of boostrap-tools hasn't updated yet.

patching /Users/Nasy/.nix/store/n36r6jhv24hwrb5v7pxxm811gwfbmyq4-bootstrap-tools/bin/xz
patching /Users/Nasy/.nix/store/n36r6jhv24hwrb5v7pxxm811gwfbmyq4-bootstrap-tools/bin/yes
ld: file not found: /usr/lib/system/libcache.dylib
builder for '/Users/Nasy/.nix/store/kwb79x0vnax2s9a6liggizdgn7kiwhpp-bootstrap-tools.drv' failed with exit code 1
cannot build derivation '/Users/Nasy/.nix/store/pzi6dc98phmzkidi6nwrqd8cphh8rar4-01-fix-ftbfs-with-glibc-2.28.patch.drv': 1 dependencies couldn't be built

[LnL7]

Closing this since initial support has been added.

I opened #104580 with details about the bootstrapping issues.

[tworitdash]

I use the following in the make file of my tool
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib
And I also get something similar. I don't know how to fix it.

ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[mroi]

This looks like a different issue. Just briefly: LD_LIBRARY_PATH is only effective at runtime and only on Linux. This issue is about link-time issues on Darwin/macOS.

Your problem looks like you need to provide the -L/usr/local/lib option to your compiler/linker, which is commonly done in Makefiles by setting LDFLAGS += -L/usr/local/lib.

If you still think this is a Nix problem, it’s probably best to open a fresh issue on GitHub.

[dxu-anduril]

Hey, I'm on the latest stable Nix version and am still getting this error when running this simple test:

[nix-shell:~]$ which clang
/nix/store/gn29d9r2j9hxvynl68skkbdngcad9fbb-clang-wrapper-7.1.0/bin/clang

[nix-shell:~]$ vim test.c

[nix-shell:~]$ echo 'int main () { return 0; }' >test.c

[nix-shell:~]$ cat test.c 
int main () { return 0; }

[nix-shell:~]$ clang test.c 
ld: file not found: /usr/lib/system/libcache.dylib for architecture x86_64
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)```

[mroi]

How did you start your Nix shell? When I start one using nix-shell --pure -p stdenv, I can successfully compile your test.c in that shell.

[dxu-anduril]

Thanks, I tried starting it like that and still received the same error:

[nix-shell:~]$ clang test.c
ld: file not found: /usr/lib/system/libcache.dylib for architecture x86_64
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)

For context, this is on the latest Big Sur.

[mroi]

Within this shell, what are your clang and ld versions? Mine:

[nix-shell:~]$ clang -v
clang version 7.1.0 (tags/RELEASE_710/final)
Target: x86_64-apple-darwin20.4.0
Thread model: posix
InstalledDir: /nix/store/7zf6wyn1wmfja7bmppb0k5vy2fq826pl-clang-7.1.0/bin

[nix-shell:~]$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-530
BUILD 19:07:38 Apr 14 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
Library search paths:
	/nix/store/3ndhndpas5zaxpa63gldf2xgcd2skd3y-libc++-7.1.0/lib
	/nix/store/9jji3hp9zj08v9zlslssihzlmlb6klhq-libc++abi-7.1.0/lib
	/nix/store/9dhwd2ibnz2gjhly1gqy58hdllxrfq2m-compiler-rt-7.1.0/lib
	/nix/store/3ndhndpas5zaxpa63gldf2xgcd2skd3y-libc++-7.1.0/lib
	/nix/store/9jji3hp9zj08v9zlslssihzlmlb6klhq-libc++abi-7.1.0/lib
	/nix/store/9dhwd2ibnz2gjhly1gqy58hdllxrfq2m-compiler-rt-7.1.0/lib
	/nix/store/wd30p2pq4lci8fk9fqldkgk1hgmwpapj-Libsystem-1238.60.2/lib
Framework search paths:
ld: no object files specified

If yours are significantly older, it’s possible that the Nixpkgs 20.09 is simply too old for proper Big Sur support in the compiler toolchain.

[dxu-anduril]

Thanks.

clang version 7.1.0 (tags/RELEASE_710/final)
Target: x86_64-apple-darwin20.4.0
Thread model: posix
InstalledDir: /nix/store/dh68ik9djfrwp9mj2pjrm51xy7y1h2m2-clang-7.1.0/bin

[nix-shell:~]$ ld -v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 00:25:07 Apr  7 2021
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
Library search paths:
	/nix/store/gr1h05c4vc7bzd9c36ffxpb0i4syd29f-libc++-7.1.0/lib
	/nix/store/cwkfvxlaz3kzlnpal3qwa252sdyib8bk-libc++abi-7.1.0/lib
	/nix/store/iqi487rh0xp0bxhgch2fw0k1imk63lri-compiler-rt-7.1.0/lib
	/nix/store/gr1h05c4vc7bzd9c36ffxpb0i4syd29f-libc++-7.1.0/lib
	/nix/store/cwkfvxlaz3kzlnpal3qwa252sdyib8bk-libc++abi-7.1.0/lib
	/nix/store/iqi487rh0xp0bxhgch2fw0k1imk63lri-compiler-rt-7.1.0/lib
	/nix/store/yv4y2bk6lljglc9ypf39akngs0qanb1r-Libsystem-osx-10.12.6/lib
Framework search paths:
ld: no object files specified

The results look comparable although I notice that mine searches Libsystem-osx instead of Libsystem. Is that potentially an issue?

[mroi]

Your linker version is too old. I think ld64-450.3 is from the Xcode 10 era. You have to move to unstable get a newer linker that works on Big Sur.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-shell-rust-hello-world-ld-linkage-issue/17381/1