K3OS Full Disk Encryption

[deeliciouscode]

Version (k3OS / kernel)

$ k3os --version: k3os version v0.22.2-k3s2r0-16-g17ddc46-dirty

$ uname --kernel-release --kernel-version: 5.4.0-88-generic #99 SMP Tue Oct 5 16:53:38 UTC 2021

Architecture

$ uname --machine: x86_64

Describe the bug

When booting from an encrypted disk, at the pivot_root stage I get a fatal error, the kernel panics.

To Reproduce

1. clone https://github.com/deeliciouscode/k3os/
2. checkout 'nolvm' (deeliciouscode@129508e)
3. in root: make
4. spin up a VM with the iso built by make
5. $ sudo /usr/libexec/k3os/install —encrypt-fs —debug /dev/sda [no yet linked to from k3os install]
6. $ sudo reboot
7. decrypt sda2
8. select one of the grub entires
9. wait for it to fail

Expected behavior

I would have expected that k3os is able to boot without an issue, since I set the flag K3OS_STATE to the device mapper, that points to the encrypted partition and should therefor work as if it was an unencrypted partition.

Actual behavior

Everything is fine up to the point where the pivot_root system call is made (https://github.com/deeliciouscode/k3os/blob/129508e353e505fb15568ee489c9bf32798eeba7/pkg/enterchroot/enter.go#L272).

I don't have deep enough kernel knowledge to easily figure out what is happening here. I thought that everything after grub should work just fine, since the K3OS_STATE Label is set on a ext4 file system, only difference beeing that it is a mapper, not a partition itself.

Additional context

We are trying to implement Full Disk Encryption for k3os and were investigating the possibilities in the last couple of weeks. After figuring out how the installation works we were able to add optional encryption to sda2, and the installation can complete successfully. The issues come in after rebooting.
Usually if one wants to encrypt the filesystem of a linux system, the disk gets partitioned, the necessary flags get set in the grub config and the mkinitfs config and then an inititfs and a grub.cfg is generated. k3os uses a different approach here. As far as I understand it, the initramfs is generated by the kernel-stage1 Dockerfile and the grub.cfg is hardcoded in the install.sh script. I think it would be preferable to have both generated during install, but especially for the initramfs I am not sure. Adding the required modules should also be possible through k3os.modules as far as I understand right now.
I am at a point where I can encrypt sda2, and am also prompted to decrypt it after rebooting. Then I am even able to select a menu entry in grub and as far as I understand the kernel is loaded properly and initrd runs successfully too. Then it takes a while and I get the error described above.
It would also be nice to get some input from one of the maintainers why the design decisions relevant here where made this way (the fact that the grub.cfg was hardcoded rather than generated for example.)

[deeliciouscode]

@dgiebert