Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos-rebuild: Allow remote building when using flakes #119540

Merged
merged 2 commits into from
May 8, 2021
Merged

nixos-rebuild: Allow remote building when using flakes #119540

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
456aa2d
nixos-rebuild: Allow remote building when using flakes
chvp Apr 15, 2021
fa827f3
nixos-rebuild: Use old-style nix command when building flake on a rem…
chvp Apr 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 53 additions & 12 deletions pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,49 @@ nixBuild() {
fi
}

nixFlakeBuild() {
if [ -z "$buildHost" ]; then
nix build "$@" --out-link "${tmpDir}/result"
readlink -f "${tmpDir}/result"
else
local attr="$1"
shift 1
local evalArgs=()
local buildArgs=()
while [ "$#" -gt 0 ]; do
local i="$1"; shift 1
case "$i" in
--recreate-lock-file|--no-update-lock-file|--no-write-lock-file|--no-registries|--commit-lock-file)
evalArgs+=("$i")
;;
--update-input)
local j="$1"; shift 1
evalArgs+=("$i" "$j")
;;
--override-input)
local j="$1"; shift 1
local k="$1"; shift 1
evalArgs+=("$i" "$j" "$k")
;;
*)
buildArgs+=("$i")
;;
esac
domenkozar marked this conversation as resolved.
Show resolved Hide resolved
done

local drv="$(nix "${flakeFlags[@]}" eval --raw "${attr}.drvPath" "${evalArgs[@]}" "${extraBuildArgs[@]}")"
if [ -a "$drv" ]; then
NIX_SSHOPTS=$SSHOPTS nix "${flakeFlags[@]}" copy --derivation --to "ssh://$buildHost" "$drv"
# The 'nix-command flakes' part in "${flakeFlags[@]}" is seen as two separate args over SSH
buildHostCmd nix --experimental-features "'nix-command flakes'" build "${buildArgs[@]}" --out-link "${tmpDir}/result" "$drv"
Copy link
Contributor

@asymmetric asymmetric Apr 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now requires that the remote host has nix.package = pkgs.nixUnstable, due to the --experimental-features flag. I think we should mention that somewhere.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

❯ ../nix/nixos-rebuild.sh switch --target-host monitor --flake ~/code/foo#monitor --use-remote-sudo
warning: Git tree '/home/asymmetric/code/foo' is dirty
building the system configuration...
warning: Git tree '/home/asymmetric/code/foo' is dirty
error: unrecognised flag '--experimental-features'
Try 'nix --help' for more information.
error: getting status of '/tmp/nixos-rebuild.u96Og6/result': No such file or directory

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that's true? As I understand it, nixos-rebuild uses it's own nix by default. I guess it could be documented for when --no-build-nix, --build-host and flakes are combined, but that seems like a rare use-case.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, posted before I saw your later comments. I wonder if the --use-remote-sudo flag is responsible?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already code that builds the latest Nix before attempting to rebuild to exactly avoid that situation, however, currently that is limited to the local machine.

nixpkgs/pkgs/os-specific/linux/nixos-rebuild/nixos-rebuild.sh

Lines 356 to 392 in 37f8212

if [[ -n $buildNix && -z $flake ]]; then
echo "building Nix..." >&2
nixDrv=
if ! nixDrv="$(nix-instantiate '<nixpkgs/nixos>' --add-root $tmpDir/nix.drv --indirect -A config.nix.package.out "${extraBuildFlags[@]}")"; then
if ! nixDrv="$(nix-instantiate '<nixpkgs>' --add-root $tmpDir/nix.drv --indirect -A nix "${extraBuildFlags[@]}")"; then
if ! nixStorePath="$(nix-instantiate --eval '<nixpkgs/nixos/modules/installer/tools/nix-fallback-paths.nix>' -A $(nixSystem) | sed -e 's/^"//' -e 's/"$//')"; then
nixStorePath="$(prebuiltNix "$(uname -m)")"
fi
if ! nix-store -r $nixStorePath --add-root $tmpDir/nix --indirect \
--option extra-binary-caches https://cache.nixos.org/; then
echo "warning: don't know how to get latest Nix" >&2
fi
# Older version of nix-store -r don't support --add-root.
[ -e $tmpDir/nix ] || ln -sf $nixStorePath $tmpDir/nix
if [ -n "$buildHost" ]; then
remoteNixStorePath="$(prebuiltNix "$(buildHostCmd uname -m)")"
remoteNix="$remoteNixStorePath/bin"
if ! buildHostCmd nix-store -r $remoteNixStorePath \
--option extra-binary-caches https://cache.nixos.org/ >/dev/null; then
remoteNix=
echo "warning: don't know how to get latest Nix" >&2
fi
fi
fi
fi
if [ -a "$nixDrv" ]; then
nix-store -r "$nixDrv"'!'"out" --add-root $tmpDir/nix --indirect >/dev/null
if [ -n "$buildHost" ]; then
nix-copy-closure --to "$buildHost" "$nixDrv"
# The nix build produces multiple outputs, we add them all to the remote path
for p in $(buildHostCmd nix-store -r "$(readlink "$nixDrv")" "${buildArgs[@]}"); do
remoteNix="$remoteNix${remoteNix:+:}$p/bin"
done
fi
fi
PATH="$tmpDir/nix/bin:$PATH"
fi

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, same thing:

❯ ../nix/nixos-rebuild.sh switch --target-host monitor --flake ~/code/foo#monitor
warning: Git tree '/home/asymmetric/code/foo' is dirty
building the system configuration...
warning: Git tree '/home/asymmetric/code/foo' is dirty
error: unrecognised flag '--experimental-features'
Try 'nix --help' for more information.
error: getting status of '/tmp/nixos-rebuild.Aa5unK/result': No such file or directory

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation for the --build-host option says that it is also built remotely.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, the nix build only happens if flakes aren't used. That's annoying.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to use old-style nix commands on the build host. It seemed like the most frictionless change to me.

buildHostCmd readlink -f "${tmpDir}/result"
else
echo "nix eval failed"
exit 1
fi
fi
}


if [ -z "$action" ]; then showSyntax; fi

Expand Down Expand Up @@ -315,8 +358,14 @@ fi

tmpDir=$(mktemp -t -d nixos-rebuild.XXXXXX)
SSHOPTS="$NIX_SSHOPTS -o ControlMaster=auto -o ControlPath=$tmpDir/ssh-%n -o ControlPersist=60"
if [ -n "$buildHost" -a -n "$flake" ]; then
buildHostCmd mkdir -p "$tmpDir"
fi

cleanup() {
if [ -n "$buildHost" -a -n "$flake" ]; then
buildHostCmd rm -rf "$tmpDir"
chvp marked this conversation as resolved.
Show resolved Hide resolved
fi
for ctrl in "$tmpDir"/ssh-*; do
ssh -o ControlPath="$ctrl" -O exit dummyhost 2>/dev/null || true
done
Expand Down Expand Up @@ -418,35 +467,27 @@ if [ -z "$rollback" ]; then
if [[ -z $flake ]]; then
pathToConfig="$(nixBuild '<nixpkgs/nixos>' --no-out-link -A system "${extraBuildFlags[@]}")"
else
outLink=$tmpDir/result
nix "${flakeFlags[@]}" build "$flake#$flakeAttr.config.system.build.toplevel" \
"${extraBuildFlags[@]}" "${lockFlags[@]}" --out-link $outLink
pathToConfig="$(readlink -f $outLink)"
pathToConfig="$(nixFlakeBuild "$flake#$flakeAttr.config.system.build.toplevel" "${extraBuildFlags[@]}" "${lockFlags[@]}")"
fi
copyToTarget "$pathToConfig"
targetHostCmd nix-env -p "$profile" --set "$pathToConfig"
elif [ "$action" = test -o "$action" = build -o "$action" = dry-build -o "$action" = dry-activate ]; then
if [[ -z $flake ]]; then
pathToConfig="$(nixBuild '<nixpkgs/nixos>' -A system -k "${extraBuildFlags[@]}")"
else
nix "${flakeFlags[@]}" build "$flake#$flakeAttr.config.system.build.toplevel" "${extraBuildFlags[@]}" "${lockFlags[@]}"
pathToConfig="$(readlink -f ./result)"
pathToConfig="$(nixFlakeBuild "$flake#$flakeAttr.config.system.build.toplevel" "${extraBuildFlags[@]}" "${lockFlags[@]}")"
fi
elif [ "$action" = build-vm ]; then
if [[ -z $flake ]]; then
pathToConfig="$(nixBuild '<nixpkgs/nixos>' -A vm -k "${extraBuildFlags[@]}")"
else
nix "${flakeFlags[@]}" build "$flake#$flakeAttr.config.system.build.vm" \
"${extraBuildFlags[@]}" "${lockFlags[@]}"
pathToConfig="$(readlink -f ./result)"
pathToConfig="$(nixFlakeBuild "$flake#$flakeAttr.config.system.build.vm" "${extraBuildFlags[@]}" "${lockFlags[@]}")"
fi
elif [ "$action" = build-vm-with-bootloader ]; then
if [[ -z $flake ]]; then
pathToConfig="$(nixBuild '<nixpkgs/nixos>' -A vmWithBootLoader -k "${extraBuildFlags[@]}")"
else
nix "${flakeFlags[@]}" build "$flake#$flakeAttr.config.system.build.vmWithBootLoader" \
"${extraBuildFlags[@]}" "${lockFlags[@]}"
pathToConfig="$(readlink -f ./result)"
pathToConfig="$(nixFlakeBuild "$flake#$flakeAttr.config.system.build.vmWithBootLoader" "${extraBuildFlags[@]}" "${lockFlags[@]}")"
fi
else
showSyntax
Expand Down