nix insists it is "unable to start any build; either increase --max-jobs or enable remote builders" even though it should be remote building

[colemickens]

Describe the bug

nix insists it is "unable to start any build; either increase --max-jobs or enable remote builders".

The problem is, I am invoking the same command that I was just invoking and running on a remote builder:

> nix --experimental-features 'nix-command flakes' build .#nixosConfigurations.azdev.config.system.build.azureImage
warning: Git tree '/home/cole/code/nixcfg' is dirty
trace: warning: The option `services.logrotate.config' defined in `/nix/store/kyxsq9xzq6lcjmh1f0wprwp52cfk1khw-source/nixos/modules/virtualisation/azure-agent.nix' has been renamed to `services.logrotate.extraConfig'.
error: --- Interrupted ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
interrupted by the user

# it was building, I noticed any tweak I need to make so I killed it

but then try again fails 30 seconds later (and 60, and 90 seconds later)

> nix --experimental-features 'nix-command flakes' \
  build \
  ".#nixosConfigurations.azdev.config.system.build.azureImage" \
   --builders 'ssh://root@nixos' \
   -j 0
warning: Git tree '/home/cole/code/nixcfg' is dirty
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to start any build; either increase '--max-jobs' or enable remote builds

I am ssh'd to nixos and there is no cpu load at all and no builds that I see going. I don't know who is wrong (my local CLI or remote daemon) but something seems wrong.

Steps To Reproduce

I don't know how easy this is to repro. I'd been doing this same process over the last few days with this same remote builder without issue...

Important Information

local machine (nix build)

> nix --version                                                                                                                                      
nix (Nix) 2.4pre20200622_334e26b

remote machine (builder)

>
[root@nixos:~]# nix --version
nix (Nix) 2.4pre20200622_334e26b

[colemickens]

Hm, on second glance, I don't have -j0 on the first line. I'm going to reclose this until I see it again...

[colemickens]

Here we go, I did have an execution that was running on the remote until I stopped it. Direct paste:

> nix --experimental-features 'nix-command flakes' build .#nixosConfigurations.azdev.config.system.build.azureImage --builders 'ssh://root@nixos' -j 0
warning: Git tree '/home/cole/code/nixcfg' is dirty
waiting for locks or build slots...
error: --- Interrupted ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
interrupted by the user

[colemickens]

If I run it with -vvvvv, I get this at the end:

building of '/nix/store/bivkvrwpz0a1ix0hfh1hdgfp3w5wh1bg-azure-image.drv': woken up
building of '/nix/store/bivkvrwpz0a1ix0hfh1hdgfp3w5wh1bg-azure-image.drv': trying to build
locking path '/nix/store/6z0jn3dh230qrj8cjvrnki0sg0dwk0gs-azure-image'
lock acquired on '/nix/store/6z0jn3dh230qrj8cjvrnki0sg0dwk0gs-azure-image.lock'
removing invalid path '/nix/store/6z0jn3dh230qrj8cjvrnki0sg0dwk0gs-azure-image'
starting build hook '/nix/store/sk9mwg4rq7s83ykinxmbmbsjbm1vwm8v-nix-2.3.7/libexec/nix/build-remote'
got 1 remote builders
considering building on remote machine 'ssh-ng://root@nixos'
hook reply is 'decline'
wait for build slot
lock released on '/nix/store/6z0jn3dh230qrj8cjvrnki0sg0dwk0gs-azure-image.lock'
building of '/nix/store/bivkvrwpz0a1ix0hfh1hdgfp3w5wh1bg-azure-image.drv': goal destroyed
killing process 4453
error: --- Error ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- nix
unable to start any build; either increase '--max-jobs' or enable remote builds

[colemickens]

Huge thank you to @Ma27 for helping me realize this is #1572.

[colemickens]

For sake of completeness, the build does work when I tell nix that the remote daemon can big-parallel and kvm.:

nix --experimental-features 'nix-command flakes' \
  build \
    ".#nixosConfigurations.azdev.config.system.build.azureImage" \
     --builders 'ssh://root@nixos x86_64-linux - - - big-parallel kvm' -j0 -vvvvvv