Initrd verify stage 2

[ElvishJerricco]

Description of changes

Adds options for verifying the system closure during boot, and signing during switch-to-configuration.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[NickCao]

Are there some background discussions on this?

[fricklerhandwerk]

This work is part of the supply chain security project funded by the STF, so indeed there is some background real-time discussion because we're trying to make the deadline. Feel free to get in touch if you have any questions and let's take it to another channel (public if you wish) and keep the PRs for technical discussion.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/boot-time-integrity-checks-for-the-nix-store/36793/1

[SuperSandro2000]

Suggested change

	ExecStart = "${nix}/bin/nix --experimental-features nix-command verify -r --store /sysroot --trusted-public-keys \"${lib.concatStringsSep " " cfg.trustedPublicKeys}\" ${if cfg.sigsNeeded == 0 then "--no-trust" else "--sigs-needed " + toString cfg.sigsNeeded} \${NIXOS_SYSTEM_CLOSURE}";
	ExecStart = "${nix}/bin/nix --extra-experimental-features nix-command verify -r --store /sysroot --trusted-public-keys \"${lib.concatStringsSep " " cfg.trustedPublicKeys}\" ${if cfg.sigsNeeded == 0 then "--no-trust" else "--sigs-needed " + toString cfg.sigsNeeded} \${NIXOS_SYSTEM_CLOSURE}";

I am not sure if it would be relevant if ca-derivations or similar are used in the closures.

[nbraud]

What do you mean? Content-address derivations have the hash in their path measure their contents, but it's not checked at boot- or access-time.

AFAIK, nix store verify still checks that the contents of CA derivations match their NAR hash, per the description:

For each path, it checks that

• its contents match the NAR hash recorded in the Nix database; and
• it is trusted, that is, it is signed by at least one trusted signing key, is content-addressed, or is built locally ("ultimately trusted").

That does lead to an interesting question, though: could an adversary manipulate the Nix db to change a CA derivation's expected hash ?

[SuperSandro2000]

Suggested change

	"/etc/systemd/system-environment-generators/nixos-environment-generator".source = "${envGenerator}/bin/nixos-environment-generator";
	"/etc/systemd/system-environment-generators/nixos-environment-generator".source = lib.getExe envGenerator;

TIL that meta.mainProgram is prefilled for writeShellScriptBin

[SuperSandro2000]

To fix the editorconfig check we probably need to add a line sort of like

nixpkgs/.editorconfig

Line 82 in 28089f8

insert_final_newline = unset

[nbraud]

Appologies if this is the wrong place to mention this, @ElvishJerricco

An alternative I experimented with (and discussed a bit with some people at CCCamp's NixOS tent this year) is using fs-verity to perform access-time verification, which solves 2 issues with boot-time verification:

• boot isn't slowed down nearly as much, since all that needs to be checked is that the filesystem has the appropriate root hashes for the Merkle tree of each file in each drv output (rather than actually hash the contents) ;
• TOCTOU (Time Of Check vs. TIme Of Use): this remains secure against an adversary tampering with storage after boot (most straightforward attack would probably be to suspend the system, write to the storage device, then resume it)

I'd be happy to share my experiences with that, and help upstream such a solution; the main issues I ran into were:

• fs-verity depends on filesystem support; in particular, ZFS does not support this feature... yet?
• builds shouldn't depend on FS support
  that could be worked around, using a userspace tool to perform the same tree hash as the kernel; libfsverity (from fsverity-utils) implements this, but its executable does not (yet?) expose the functionality

[ElvishJerricco]

@nbraud It would probably be best to share that here: https://discourse.nixos.org/t/boot-time-integrity-checks-for-the-nix-store/36793

But the short version of my findings is that you can't actually verify that files are in the right locations with fs-verity, so e.g. it doesn't protect against the bash and systemd binaries being swapped, causing PID 1 to be a shell.

[nbraud]

@nbraud It would probably be best to share that here: https://discourse.nixos.org/t/boot-time-integrity-checks-for-the-nix-store/36793

OK, let's take it to Discourse.
P.S. Having issues logging into Discourse, but I'll be home in a few days so I'll see then.