nixos-rebuild: Allow remote building when using flakes

[chvp]

Motivation for this change

Remotely building hosts when using flakes should be possible. See also #119448, https://discourse.nixos.org/t/building-a-flake-based-nixos-system-remotely/11309/4, and https://nixos.wiki/wiki/Flakes#Using_nix_flakes_with_NixOS

Things done

I rebuilt my system with this PR, and then used nixos-rebuild to build a host on itself, build a host on another remote host and build the local host on a remote host.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/building-a-flake-based-nixos-system-remotely/11309/5

[asymmetric]

This now requires that the remote host has nix.package = pkgs.nixUnstable, due to the --experimental-features flag. I think we should mention that somewhere.

[asymmetric]

❯ ../nix/nixos-rebuild.sh switch --target-host monitor --flake ~/code/foo#monitor --use-remote-sudo
warning: Git tree '/home/asymmetric/code/foo' is dirty
building the system configuration...
warning: Git tree '/home/asymmetric/code/foo' is dirty
error: unrecognised flag '--experimental-features'
Try 'nix --help' for more information.
error: getting status of '/tmp/nixos-rebuild.u96Og6/result': No such file or directory

[chvp]

I don't think that's true? As I understand it, nixos-rebuild uses it's own nix by default. I guess it could be documented for when --no-build-nix, --build-host and flakes are combined, but that seems like a rare use-case.

[chvp]

Sorry, posted before I saw your later comments. I wonder if the --use-remote-sudo flag is responsible?

[hmenke]

There is already code that builds the latest Nix before attempting to rebuild to exactly avoid that situation, however, currently that is limited to the local machine.

nixpkgs/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh

Lines 356 to 392 in 37f8212

	if [[ -n $buildNix && -z $flake ]]; then
	echo "building Nix..." >&2
	nixDrv=
	if ! nixDrv="$(nix-instantiate '<nixpkgs/nixos>' --add-root $tmpDir/nix.drv --indirect -A config.nix.package.out "${extraBuildFlags[@]}")"; then
	if ! nixDrv="$(nix-instantiate '<nixpkgs>' --add-root $tmpDir/nix.drv --indirect -A nix "${extraBuildFlags[@]}")"; then
	if ! nixStorePath="$(nix-instantiate --eval '<nixpkgs/nixos/modules/installer/tools/nix-fallback-paths.nix>' -A $(nixSystem) | sed -e 's/^"//' -e 's/"$//')"; then
	nixStorePath="$(prebuiltNix "$(uname -m)")"
	fi
	if ! nix-store -r $nixStorePath --add-root $tmpDir/nix --indirect \
	--option extra-binary-caches https://cache.nixos.org/; then
	echo "warning: don't know how to get latest Nix" >&2
	fi
	# Older version of nix-store -r don't support --add-root.
	[ -e $tmpDir/nix ] || ln -sf $nixStorePath $tmpDir/nix
	if [ -n "$buildHost" ]; then
	remoteNixStorePath="$(prebuiltNix "$(buildHostCmd uname -m)")"
	remoteNix="$remoteNixStorePath/bin"
	if ! buildHostCmd nix-store -r $remoteNixStorePath \
	--option extra-binary-caches https://cache.nixos.org/ >/dev/null; then
	remoteNix=
	echo "warning: don't know how to get latest Nix" >&2
	fi
	fi
	fi
	fi
	if [ -a "$nixDrv" ]; then
	nix-store -r "$nixDrv"'!'"out" --add-root $tmpDir/nix --indirect >/dev/null
	if [ -n "$buildHost" ]; then
	nix-copy-closure --to "$buildHost" "$nixDrv"
	# The nix build produces multiple outputs, we add them all to the remote path
	for p in $(buildHostCmd nix-store -r "$(readlink "$nixDrv")" "${buildArgs[@]}"); do
	remoteNix="$remoteNix${remoteNix:+:}$p/bin"
	done
	fi
	fi
	PATH="$tmpDir/nix/bin:$PATH"
	fi

[asymmetric]

No, same thing:

❯ ../nix/nixos-rebuild.sh switch --target-host monitor --flake ~/code/foo#monitor
warning: Git tree '/home/asymmetric/code/foo' is dirty
building the system configuration...
warning: Git tree '/home/asymmetric/code/foo' is dirty
error: unrecognised flag '--experimental-features'
Try 'nix --help' for more information.
error: getting status of '/tmp/nixos-rebuild.Aa5unK/result': No such file or directory

[chvp]

The documentation for the --build-host option says that it is also built remotely.

[chvp]

Ah, the nix build only happens if flakes aren't used. That's annoying.

[chvp]

I changed it to use old-style nix commands on the build host. It seemed like the most frictionless change to me.

[asymmetric]

@chvp I think it would be a bit easier to review if you didn't force-push, so we could look at the diff in each commit ❤️

[hmenke]

@edolstra @worldofpeace @obadz @volth @domenkozar @Mic92
Could someone please review this? Having working remote builds with flakes would be a nice feature for 21.05. (You were pinged because you previously committed to this file)

[chvp]

Just wanted to mention that I've been using this to build my systems remotely since I created this PR without any issues.

[domenkozar]

This PR doesn't change behavior if host is not defined and a few people reported success with it, so let's merge it.

[jonringer]

This PR doesn't change behavior if host is not defined and a few people reported success with it, so let's merge it.

This wasn't correct because of the following lines:

if [ -z "$buildHost" -a -n "$targetHost" ]; then
    buildHost="$targetHost"
fi
if [ "$targetHost" = localhost ]; then
    targetHost=
fi
if [ "$buildHost" = localhost ]; then
    buildHost=
fi

A previous nixos-rebuild --target-host remote.host --flake <flake> would build the outputs locally, then nix-copy them over. New behavior would be instantiate the .drv and nix copy the entire .drv closure.

The consequences was that 2000-4400 store paths would need be transferred instead of the previous 100-400 realised paths.