dm-verity signed /boot

[DemiMarie]

Is your feature request related to a problem? Please describe.
Verifying signatures on individual files in /boot is error-prone and slow.

Describe the solution you'd like
Use dm-verity and multiple possible /boot partitions. This allows atomic updates of /boot by replacing the entire partition. Heads will automatically choose the newest version.

Describe alternatives you've considered
#1599

Additional context
dm-verity is used by most other verified boot schemes.

[tlaurion]

@DemiMarie

Community is pushing the other way around, wanting more output on files being hashed/verified as #1369, not having Heads stops because some OS changed meta-data only under /boot and expect OSes to have two /boot partitions (who does that? Is there where QubesOS wants to go for Q4.3? IS this a practical solution that we can expect OS distributions to follow in their default installation partition schemes? Therefore: how is this a practical suggestion?)

As per past discussions, the ideal here would be for OSes to sign their kernel/initrd/UKI, and once there is progress there, have Heads validate those as an additional security mechanism first, then switch to that.

What Heads currently does

Recap on what Heads does, here and was improved in the past years in terms of boot integrity validation/authentication

• tree of /boot dirs/files tree to list existing dirs/files and be able to discover (to diff new files which otehrwise cannot be verified against only with hashes): /boot/kexec_tree.txt
• hash of /boot files that are checked against prior of booting default boot option and warns user if changed: /boot/kexec_hashes.txt
• default boot params part of trusted kexec call per default signed op: /boot/kexec_default.1.txt
• default boot files hashes per detached signed op: /boot/kexec_default_hashes.txt
• default boot+TPM DUK setup created known LUKS devices: /boot/kexec_key_devices.txt
• TPM2 primary handle hash per TPM ownership/last TOTP/HOTP sealed op: /boot/kexec_primhdl_hash.txt
• default+TPM DUK setup initrd crypptab override (to pass secret.key to OS) : /boot/kexec_initrd_crypptab_overrides.txt
• detached signed signature of everything /boot/kexec_*.txt verified against each default boot: /boot/kexec.sig

---

And then as described under qubes forum: https://forum.qubes-os.org/t/verifying-installation/11739/4.

Simulation from dom0

sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig - && cd /boot && sudo sha256sum -c /boot/kexec_hashes.txt

Output without dom0 having public key:

(2)$ sha256sum $(find /boot/kexec*.txt) | gpg --verify /boot/kexec.sig -
gpg: Signature made Tue 26 Mar 2024 04:03:48 PM EDT
gpg:                using RSA key 575F80D1599EA6D2C70AA9A19A53E1BB3FF00461
gpg: Can't check signature: No public key

And then, again from dom0:

$ cd /boot
$ time sudo sha256sum -c kexec_hashes.txt 
./config-6.1.62-1.qubes.fc37.x86_64: OK
./System.map-6.1.62-1.qubes.fc37.x86_64: OK
./xen-4.17.3.config: OK
./vmlinuz-6.1.62-1.qubes.fc37.x86_64: OK
./grub2/themes/qubes/progress_bar_c.png: OK
./grub2/themes/qubes/unifont-bold-16.pf2: OK
./grub2/themes/qubes/unifont-regular-14.pf2: OK
./grub2/themes/qubes/unifont-regular-32.pf2: OK
./grub2/themes/qubes/progress_bar2_c.png: OK
./grub2/themes/qubes/qubes.png: OK
./grub2/themes/qubes/unifont-regular-16.pf2: OK
./grub2/themes/qubes/progress_bar_hl_c.png: OK
./grub2/themes/qubes/theme.txt: OK
./grub2/grub.cfg: OK
./grub2/grubenv: OK
./initramfs-6.1.62-1.qubes.fc37.x86_64.img: OK
./symvers-6.1.75-1.qubes.fc37.x86_64.gz: OK
./vmlinuz-6.1.75-1.qubes.fc37.x86_64: OK
./initramfs-6.1.75-1.qubes.fc37.x86_64.img: OK
./xen-4.17.3.gz: OK
./System.map-6.1.75-1.qubes.fc37.x86_64: OK
./config-6.1.75-1.qubes.fc37.x86_64: OK
./symvers-6.1.62-1.qubes.fc37.x86_64.gz: OK

real	0m0.114s
user	0m0.086s
sys	0m0.026s

---

Answers

Is your feature request related to a problem? Please describe.
Verifying signatures on individual files in /boot is error-prone and slow.

As explained above, there seems to be underlying incomprehension here, most probably from a lack of hands-on experience prior of speaking resulting in impractical proposition.

Once again, Heads testing is possible under a qube from built and ran qemu-coreboot-whiptail-tpm2 ofr certified hardware, coming with TRACE and DEBUG calls output so understand how things work doesn't get in the way, if not ideally on real hardware?

Hands-on, practical suggestions, applicable on what distributions actually deploy would help (and also understand why nested virt is needed for qemu testing under QubesOS, otherwise sloooooooooooooooow). Otherwise have non-qubes testing machine to run Heads under KVM and understand what is happening under the hood.

Who deploys dm-verity today anyway?

Slow?

The detached signature is on hashes files and kexec related pre-boot config files checked by detached signature, not on files directly. But even if the files were checked, that wouldn't change much in time of perf on typical deployments (@saper : haven't found yet a solution for your use case outside of seperating ISO partition, which QubesOS still cannot boot from HD partitions last time I checked, which is QubesOS/qubes-issues#8054).

Just like distributions are getting away of ISO detached signatures (Fedora 19?) in favour of hash files being detached signed and the verification process being first on validating the hash file integrity and then verifying the integrity of the ISO file through hashing functions directly, this is what is being done here.

There is no point validating the files hashes if the gllobal hash file integrity is invalid against detached signed global hash file : exiting early there is better than testing each file integrity against GPG: this is why it's done like that now where more verbose output is desired, not less.

Error-prone?

If you're talking about FS accepted chars (which could go wrong), that was implemented at the same time of that tree file and the parsing of files discovered on the /boot partition, while the same code is used for root integrity feature, now supported under LUKS container (but yet again: no point here doing that for QubesOS because QubesOS/qubes-issues#4371 (comment))

Code to be improved, if you want to review that now, is under print_tree and assert_signable that were added under #1262

---

Discussions

Q: What would be the advantages of your proposed solution as opposed to #1599 or #1615 ?
#1615 is the real solution, but would require QubesOS to attack the problem and not defer to Fedora and/or to contribute fix to Fedora. I looked into UKI as also implied under #1550 (comment) but no distro is actually enforcing it today, deferring to the end user the task of signing UKI, which would also better fix this issue then proposed solution.

I would also defer the problem to having /boot encrypted under LUKS but then again, using dm-verity would come with a lot of other issues since opening /boot is not even always done in R/O and touching FS metadata will also make dm-verity fail. So what is the added value for the end user at this point?

dm-verity is used by most other verified boot schemes
Who does it? Where can I "most other verified boot schemes" using dm-verity?

QubesOS still need to seperate dom0 rootfs writing, which AFAIK depends on

• Externalizing logging to sys-log (https://forum.qubes-os.org/t/working-guide-for-creating-a-logging-qube-i-e-sys-log-wip/6587/2)
• Having dom0 stop writing to rootfs, see Secure boot support QubesOS/qubes-issues#4371 (comment)
• Switch from fedora to silverblue for dom0 layer?

[DemiMarie]

Who does it? Where can I "most other verified boot schemes" using dm-verity?

dm-verity is used by Android, ChromeOS, and I believe many embedded systems. Apple uses something similar (Sealed System Image) in macOS and iOS.

That said, if you think this should be closed in favor of UKI (which is what I think Qubes OS is moving to for secure boot support) and automatic signing on updates, that’s fine.

[tlaurion]

Who does it? Where can I "most other verified boot schemes" using dm-verity?

dm-verity is used by Android, ChromeOS, and I believe many embedded systems.

Android also seperates SYSTEM partition from config and deploys this with things embedded in firmware with VBOOT to enforce this. Doing so, they also permit to "Factory Reset" user partition, which goes along the lines of my suggestion of going Silverblue way.

Apple uses something similar (Sealed System Image) in macOS and iOS.

Have not followed this, but pretty sure the idea is for OS to be closely embedded in firmware doing a similar thing than Android above.

That said, if you think this should be closed in favor of UKI (which is what I think Qubes OS is moving to for secure boot support) and automatic signing on updates, that’s fine.

It's not important what I think here outside of implementing something that doesn't lock down users of their system because the OS doesn't comply with what is attempted in the bootloader. Even less implementing something that no distro currently uses or at least plan to use.

As said above and in prior discussions, as long as OSes do not implement that something suggested, that that something can be leveraged and enforced in firmware today, I just wanted to restate once again that those discussions should happen upstream and be referenced downstream (here) for follow up with proper references otherwise those discussions are purely idealistic and philosophical at best. Heads tries to implement what is readily available and enforceable, today.

None of the suggested solutions, outside of the rejected git repo under /boot, can actually tell what changed, and even less say if those changes are as prescribed by the OS. UKI will answer to the last point but won't help for what the OS deploys as well under /boot nor rootfs unless something better is made to properly authenticate the files, otherwise Heads can only hide /boot under LUKS but that won't help anything in the realms of investigating the changes nor guiding the user into auditing the changes at all.

As opposed to Android or Apple perspectives/implementation referred here, which control not only the OS states but also the firmware, implementing dm-verity as suggested would be a regression, not an improvement over current situation. OK, /boot changed. Now what?

What changed? When? By who? Why?

---

Heads can now inspect rootfs and hash content there on demand and also report on changes. We cannot use that on QubesOS today either. My suggestion there was to somewhat trust the OS enough to at least use it's reported dnf database for hashes validation of what is deployed under /boot and be able to blame/point to the date of the change and the update that was done on that date and also validating that the states that changed under /boot were confirmed by the OS, which is responsible to update the /boot partition.
@JonathonHall-Purism said OS state should not be trusted. So I'm out of ideas outside of waiting for UKI but that solution would be incomplete as well.

Other ideas?

[tlaurion]

Original discussion https://groups.google.com/g/qubes-devel/c/hG93VcwWtRY

[tlaurion]

Related QubesOS/qubes-issues#9146

[marmarek]

I'm not sure about other OSes supported by Heads, but currently there are no plans for having separate /boot (dm-verity protected or not) partitions for different kernel versions in Qubes OS. So, unless some other OS used with Heads would support such feature, IMO having this in Heads makes no sense.

[DemiMarie]

I’ll close this issue, then. It’s ultimately a hack anyway, and UKIs (with a hardened FAT filesystem parser) would be a much better choice.