Nix-daemon process leaking on nix-darwin Mac build machine

[johnalotoski]

With nix-darwin on a mac-build machine, the nix-daemon appears to be spawned repeatedly and not close the process out during builds. After some time, when the ulimit process limits have been hit, the build machine yields resource errors, such as:

Could not spawn trampoline /usr/libexec/xpcproxy: 35: Resource temporarily unavailable
Deferred spawn of service failed: 35: Resource temporarily unavailable

Some diagnostic commands below:

-sh-3.2# nix-channel --list
darwin https://github.com/input-output-hk/nix-darwin/archive/master.tar.gz
nixpkgs https://nixos.org/channels/nixos-19.03

# nix run nixpkgs.nix-info -c nix-info -m
 - system: `"x86_64-darwin"`
 - host os: `Darwin 18.7.0, macOS 10.14.6`
 - multi-user?: `no`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.1.3`
 - channels(root): `"darwin, nixpkgs-19.03.173672.5f7eae4bbb1"`
 - channels(nixos): `"darwin, nixpkgs-19.03.173672.5f7eae4bbb1"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

-sh-3.2# ps aux | grep nix-daemon
root            86871   0.0  0.3  4327444  17056   ??  Ss    3:34PM   0:00.50 /nix/var/nix/profiles/default/bin/nix-daemon
root            84285   0.0  0.3  4335860  18944   ??  Ss    3:33PM   0:00.61 /nix/var/nix/profiles/default/bin/nix-daemon
root            71620   0.0  0.2  4328276  14804   ??  Ss    3:31PM   0:00.22 /nix/var/nix/profiles/default/bin/nix-daemon
root            56956   0.0  0.3  4327500  17756   ??  Ss    3:29PM   0:00.45 /nix/var/nix/profiles/default/bin/nix-daemon
root            49187   0.0  0.3  4335848  17736   ??  Ss    3:27PM   0:00.31 /nix/var/nix/profiles/default/bin/nix-daemon
...

(gdb) thread apply all bt

Thread 2 (Thread 0x1003 of process 77402):
#0  0x00007fff606f49de in __ulock_wait () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff607b56de in _pthread_join () from /usr/lib/system/libsystem_pthread.dylib
#2  0x000000010ec976e8 in std::__1::thread::join() () from /nix/store/g5avn9k0sdn51rdwji3maikkl7a4sqaf-libc++-5.0.2/lib/libc++.1.0.dylib
#3  0x000000010e9c49a0 in nix::CurlDownloader::~CurlDownloader() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixstore.dylib
#4  0x000000010e97b111 in std::__1::shared_ptr<nix::Store>::~shared_ptr() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixstore.dylib
#5  0x00007fff606633cf in ?? ()
#6  0x00007fb009400000 in ?? ()
#7  0x0000000000000008 in ?? ()
#8  0x000000000940bb10 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 1 (Thread 0xd03 of process 77402):
#0  0x00007fff606f4f06 in __psynch_mutexwait () from /usr/lib/system/libsystem_kernel.dylib
#1  0x00007fff607b1d52 in _pthread_mutex_firstfit_lock_wait () from /usr/lib/system/libsystem_pthread.dylib
#2  0x00007fff607af4cd in _pthread_mutex_firstfit_lock_slow () from /usr/lib/system/libsystem_pthread.dylib
#3  0x000000010ec83749 in std::__1::mutex::lock() () from /nix/store/g5avn9k0sdn51rdwji3maikkl7a4sqaf-libc++-5.0.2/lib/libc++.1.0.dylib
#4  0x000000010ebce483 in nix::InterruptCallbackImpl::~InterruptCallbackImpl() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixutil.dylib
#5  0x000000010ebce44e in nix::InterruptCallbackImpl::~InterruptCallbackImpl() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixutil.dylib
#6  0x000000010e9ba733 in nix::CurlDownloader::workerThreadMain() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixstore.dylib
#7  0x000000010e9b958a in nix::CurlDownloader::workerThreadEntry() () from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixstore.dylib
#8  0x000000010e9b94ed in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, nix::CurlDownloader::CurlDownloader()::{lambda()#1}> >(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, nix::CurlDownloader::CurlDownloader()::{lambda()#1}>) ()
   from /nix/store/dkjlfkrknmxbjmpfk3dg4q3nmb7m3zvk-nix-2.1.3/lib/libnixstore.dylib
#9  0x00007fff607b12eb in _pthread_body () from /usr/lib/system/libsystem_pthread.dylib
#10 0x00007fff607b4249 in _pthread_start () from /usr/lib/system/libsystem_pthread.dylib
#11 0x00007fff607b040d in thread_start () from /usr/lib/system/libsystem_pthread.dylib
#12 0x0000000000000000 in ?? ()

[roberth]

This happened on a Hercules CI agent machine too.

ps aux | grep nix-daemon | wc -l
792

[RocketPuppy]

I'll try to find more diagnostic info if it happens again, but I'm seeing this behavior (resource limit reached) on two 20.03 NixOS machines. One is a laptop that gets tweaked and rebuilt frequently. The other is a computer managed by NixOps.

[johnalotoski]

FWIW, here's a small module with an ugly bash hack to expire pids so they don't cause that issue:
https://github.com/input-output-hk/ci-ops/blob/master/nix-darwin/modules/expire-pids.nix
https://github.com/input-output-hk/ci-ops/blob/master/nix-darwin/services/expire-pids.nix

[stale]

I marked this as stale due to inactivity. → More info