-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
darwin: nix-daemon crashes due to OBJC_DISABLE_INITIALIZE_FORK_SAFETY #2523
Comments
Why is Objective C even relevant to |
Yeah, I don't understand this either. I'm running 10.14 on my new machine and don't have any problems with builds or the daemon. |
@periklis can you run your daemon with |
Here is the log: https://gist.github.com/periklis/a4357cfd21cd67dafa9122f9ca119592 Is it maybe because nix-daemon is run by launchctl that in turn can be an objc software? |
I've hit the same thing. However... as @copumpkin noticed the problem is deep blow the nix stack and in fact is related to some awkward security features introduced somewhere in High Sierra, setting |
We got a bit closer to figuring out what's going on. The released nix binaries work fine, however using a build from nixpkgs master/unstable (on 10.14.1?) causes this problem. Either this is caused by the new CoreFoundation from swift-corelibs or something else is pulling in the Objective-C runtime. With changes like NixOS/nixpkgs#29785 libraries like libcurl have become pretty bloated IMHO, pulling in eg. kerberos in by default (not only for curlFull). One of those could also be causing this. |
yeah, my setup has a managed nix binary by nix-darwin |
First I thought it was a problem there somehow, but using the same binary from the installer there works fine. { config, lib, pkgs, ... }:
{
nix.package = lib.toDerivation /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3 // { version = "2.1.3"; };
} |
I updated my machine but still can't reproduce this for some reason. |
@LnL7 10.14.1 or 10.14.0? i am still on later. |
I can reproduce this now. A bunch of frameworks are definitively getting loaded and I'm not sure why yet, however this doesn't seem to be anything if I compare it with the working version. @@ -1,4 +1,4 @@
-dyld: loaded: good/bin/nix-daemon
+dyld: loaded: bad/bin/nix-daemon
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libsodium-1.0.16/lib/libsodium.23.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nix-2.2pre6526_9f99d624/lib/libnixexpr.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-boehm-gc-8.0.0/lib/libgc.1.dylib
@@ -50,7 +50,7 @@
dyld: loaded: /usr/lib/libc++.1.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-xz-5.2.4/lib/liblzma.5.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-bzip2-1.0.6.0.1/lib/libbz2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libcrypto.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libcrypto.1.0.0.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-brotli-1.0.7-lib/lib/libbrotlienc.1.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-brotli-1.0.7-lib/lib/libbrotlidec.1.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-boost-1.67_0/lib/libboost_context.dylib
@@ -62,15 +62,15 @@
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libiconv-osx-10.11.6/lib/libiconv-nocharset.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libiconv-osx-10.11.6/lib/libcharset.1.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libc++abi-5.0.2/lib/libc++abi.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.24.0/lib/libsqlite3.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-sqlite-3.26.0/lib/libsqlite3.0.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-curl-7.59.0/lib/libcurl.4.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-transfer.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-s3.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-aws-sdk-cpp-1.6.52/lib/libaws-cpp-sdk-core.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-zlib-1.2.11/lib/libz.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.34.0-lib/lib/libnghttp2.14.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.35.0-lib/lib/libnghttp2.14.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libssh2-1.8.0/lib/libssh2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libssl.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libssl.1.0.0.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libgssapi_krb5.2.2.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-Libsystem-osx-10.11.6/lib/libresolv.9.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-swift-corefoundation/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
@@ -81,10 +81,10 @@
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-ICU-osx-10.10.5/lib/libicucore.A.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-curl-7.62.0/lib/libcurl.4.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libxml2-2.9.8/lib/libxml2.2.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.34.0-lib/lib/libnghttp2.14.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-nghttp2-1.35.0-lib/lib/libnghttp2.14.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libssh2-1.8.0/lib/libssh2.1.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libssl.1.0.0.dylib
-dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2p/lib/libcrypto.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libssl.1.0.0.dylib
+dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-openssl-1.0.2q/lib/libcrypto.1.0.0.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libgssapi_krb5.2.2.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libkrb5.3.3.dylib
dyld: loaded: /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-libkrb5-1.15.2/lib/libk5crypto.3.1.dylib
@@ -238,5 +238,7 @@
dyld: loaded: /usr/lib/libcmph.dylib
dyld: loaded: /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
dyld: loaded: /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
-dyld: loaded: /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
+dyld: loaded: /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
dyld: loaded: /usr/lib/libxslt.1.dylib
+objc[38361]: +[__NSPlaceholderArray initialize] may have been in progress in another thread when fork() was called.
+objc[38361]: +[__NSPlaceholderArray initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. |
I think I found the culprit, something dlopen's libnetwork.dylib which then pulls in lots of system stuff and frameworks.
EDIT: possibly curl -> libresolv -> Libinfo |
Found some more information the dlopen comes from curl. However this is not something new so I'm not sure what changed or how to fix it.
|
I made an example program to reproduce the behaviour (without forking). It looks like some kind of fallback that happens when no ipv6 address can be resolved for the host. https://gist.github.com/LnL7/5c43975144c0a9a0f5de74cc155a01d5 |
I'm having the same (or similar?) experience on 10.14.2 fwiw. Edit: I'm also having some weird errors with substituters, so I can't neither download cached binaries, nor build with |
I think I might have found the problem thanks to NixOS/nixpkgs#41312 (comment)
A build from master still seems to have the same problem, while this does not. with import <nixpkgs> {};
let
curlFix = curl.overrideAttrs (drv: {
configureFlags = drv.configureFlags or [] ++ ["--disable-threaded-resolver"];
});
in
nix.override {
curl = curlFix;
aws-sdk-cpp = aws-sdk-cpp.override { curl = curlFix; };
} EDIT: While this definitively improves the situation there are still cases where problems occur. For some bizarre reason substituting from my local cache still fails while cache.nixos.org now works consistently. |
Do we think this is a curl bug or a Nix bug? It could be the way that Nix is downloading with curl is wrong. I just started hitting this after updating to 10.14.3. |
This is a workaround for NixOS#2523.
Since macOS 10.14 this has become an error, causing problems if the nix-daemon loads nix during substitution (this is a forked process). Workaround for NixOS#2523.
That's possible. But since this only started to happen with a recent nixpkgs/curl version I suspect it's curl or libresolv itself. |
Had to write a script to add the OBJC_DISABLE_INITIALIZE_FORK_SAFETY to coworker's nix-daemons when this started happening. In case anyone's curious, you can use PlistBuddy and a plist file to do this in a bash script pretty easily:
set -e
set -o pipefail
NIX_DAEMON_PLIST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
set -x
sudo launchctl unload $NIX_DAEMON_PLIST
while ps -ax | grep -v grep | grep "nix-daemon"
echo "Waiting for nix-daemon to stop..."
sleep 1 # not great but generally the daemon closes very fast if you're not using it.
end
sudo /usr/libexec/PlistBuddy -c "merge $PWD/add-obj-c-fork-env.plist EnvironmentVariables" $NIX_DAEMON_PLIST || true
sudo launchctl load -w $NIX_DAEMON_PLIST |
Since macOS 10.14 this has become an error, causing problems if the nix-daemon loads nix during substitution (this is a forked process). Workaround for NixOS#2523. (cherry picked from commit 8ac1130)
@corps One other option, if you use |
Ran into the same problem and tried the workaround script, but I had to fix it. It doesn't need the separate plist file anymore: set -x
set -e
set -o pipefail
NIX_DAEMON_PLIST=/Library/LaunchDaemons/org.nixos.nix-daemon.plist
sudo launchctl unload $NIX_DAEMON_PLIST
while ps -ax | grep -v grep | grep "nix-daemon"; do
echo "Waiting for nix-daemon to stop..."
sleep 1 # not great but generally the daemon closes very fast if you're not using it.
done
sudo /usr/libexec/PlistBuddy \
-c "Add :EnvironmentVariables:OBJC_DISABLE_INITIALIZE_FORK_SAFETY string YES" \
$NIX_DAEMON_PLIST || true
sudo launchctl load -w $NIX_DAEMON_PLIST |
I still get this issue, and the OBJC_DISABLE_INITIALIZE_FORK_SAFETY plist fix does not fix it for me. This only happens in multi-user mode, regular nix install works fine. Heres my nix-info from single user mode:
|
Are you sure the daemon is running with that variable?
|
EDIT 2: My issue was unrelated. I had some kind of mess left over from a previous nix install that prevented the default profile from being set up properly. Click to see original commentYes, I do see exactly that. I also see that variable on a fresh install, without even running the fix (I'm assuming this made it into the latest release). I sometimes see different error messages: sometimes it only says ➜ ~ nix-shell -p nix-info --run "nix-info -m"
error (ignored): unable to download 'https://cache.nixos.org/jl50iznh81znnf2jw9fy32hs9m0s577d.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
error: unexpected end-of-file
➜ ~ nix-shell -p nix-info --run "nix-info -m"
error (ignored): unable to download 'https://cache.nixos.org/7q9s8jlnz3m2079960jpbcpg4nsvrk1m.narinfo': Problem with the SSL CA cert (path? access rights?) (77)
error: unexpected end-of-file
➜ ~ nix-shell -p nix-info --run "nix-info -m"
error: unexpected end-of-file A full ➜ ~ cat /Library/LaunchDaemons/org.nixos.nix-daemon.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>EnvironmentVariables</key>
<dict>
<key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
<string>YES</string>
</dict>
<key>Label</key>
<string>org.nixos.nix-daemon</string>
<key>KeepAlive</key>
<true/>
<key>RunAtLoad</key>
<true/>
<key>Program</key>
<string>/nix/store/87i4fp46jfw9yl8c7i9gx75m5yph7irl-nix-2.2.2/bin/nix-daemon</string>
<key>StandardErrorPath</key>
<string>/var/log/nix-daemon.log</string>
<key>StandardOutPath</key>
<string>/dev/null</string>
</dict>
</plist>
➜ ~ launchctl print system/org.nixos.nix-daemon | grep OBJC_DISABLE_INITIALIZE_FORK_SAFETY
OBJC_DISABLE_INITIALIZE_FORK_SAFETY => YES Here's what I've seen in
Also, @LnL7 thank you for the quick response. I much appreciate it. I think the cert problem means that I'm commenting in the wrong issue. Do you know what the right place to triage this would be? Please let me know if there is some other log I can provide to help diagnose the issue. EDIT: I think its related to the fact that in the multi-user install, my ~/.nix-profile is a dangling symlink: single-user install:
multi-user install just doesn't have the
And so the SSL certs in |
In addition to what @ldub said, you'll see
as well if |
I've just had a very similar issue to @ldub, the difference being that my nix-profile was completely empty, not even the channel files. |
@ldub How did you fix your issue? I see you describe your problem but there's no details on the fix |
@webframp my issue was that ~/.nix-profile was a dangling symlink. Unlikely to be related to you problem. |
I marked this as stale due to inactivity. → More info |
As far as I know, this is still relevant for |
I marked this as stale due to inactivity. → More info |
On macOS 10.14 i observe lately in my system.log that
nix-daemon
crashes due to the apple changes onfork
since 10.13. The following message comes in regurarly:Related discussion in nixpkgs for other packages: NixOS/nixpkgs#44160
Related info: http://sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html
Do we need to update the SDK dependency for our buildsystem?
The text was updated successfully, but these errors were encountered: