Unable to create bootable (bios) Debian image

[whot]

I'm unable to create any qemu-bootable Debian image using BootProtocols=bios.

mkosi.default file looks like this:

[Distribution]
Distribution=debian
Release=testing

[Output]
Format=gpt_ext4
Bootable=yes
BootProtocols=bios

[Partitions]
RootSize=2G

Same bug appears for Debian stable/unstable and the recent Ubuntus (tested the last two, I think). In all cases, the issue is that there is no initrd file in /boot which then unsurprisingly fails during boot. The same mkosi.default file with Fedora 33 as distribution works just fine.

Host system is Fedora 33, reproducible with:

$ sudo mkosi --force
$ qemu-system-x86_64 -machine accel=kvm \
                       -smp 2 -m 1024 \
                       -drive format=raw,file=image.raw \
                       -device virtio-net-pci,netdev=net0 \
                       -netdev user,id=net0,hostfwd=tcp::5555-:22 \
                       -display sdl \
                       -serial file:console.out

git bisect says this started with 46ceca8 which (handwavy guess) could explain why the initrd never appears, maybe the partition isn't mounted correctly?

[behrmann]

Thanks for report and the bisect, @whot. This is an excellent starting point to find the reason.

[DaanDeMeyer]

I get a kernel panic with this config, can you confirm you get the same error?

[    0.254608] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.256038] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.9.0-4-amd64 #1 Debian 5.9.11-1
[    0.257113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ArchLinux 1.14.0-1 04/01/2014
[    0.258631] Call Trace:
[    0.258995]  dump_stack+0x6b/0x83
[    0.259462]  panic+0x101/0x2d7
[    0.259891]  mount_block_root+0x304/0x313
[    0.260478]  prepare_namespace+0x136/0x165
[    0.261054]  kernel_init_freeable+0x1ef/0x1fa
[    0.261659]  ? rest_init+0xaa/0xaa
[    0.262135]  kernel_init+0xa/0x106
[    0.262613]  ret_from_fork+0x22/0x30
[    0.263152] Kernel Offset: 0x800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[    0.264603] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

[whot]

Yep, same error. And you probably have better tools, but I used this to check /boot:

offset=$(fdisk -l image.raw | grep image.raw2 | cut -d ' ' -f 3)
sudo mount -o ro,loop,offset=$((offset * 512)) image.raw loop/
sudo ls loop/boot

and weep at the lack of an initrd file :)

[whot]

I think the git bisect was a false positive, or maybe there are other bugs at play. 925a009 introduced this code:

+ if not do_run_build_script and args.bootable and args.with_unified_kernel_images:
+        # Disable dracut postinstall script for this apt-get run.
+        env['INITRD'] = 'No'

I'm guessing this is a pretty good hint for "don't create an initrd". Removing the env line makes dracut generate the initrd with the above example.

Before:

Setting up dracut-core (050+65-1) ...
Setting up dracut (050+65-1) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.9.0-4-amd64 (5.9.11-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.9.0-4-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.9.0-4-amd64
Setting up linux-image-amd64 (5.9.11-1) ...
Processing triggers for libc-bin (2.31-5) ...
Processing triggers for dracut (050+65-1) ...

After:

Setting up dracut-core (050+65-1) ...
Setting up dracut (050+65-1) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.9.0-4-amd64 (5.9.11-1) ...
I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.9.0-4-amd64
I: /initrd.img.old is now a symlink to boot/initrd.img-5.9.0-4-amd64
I: /vmlinuz is now a symlink to boot/vmlinuz-5.9.0-4-amd64
I: /initrd.img is now a symlink to boot/initrd.img-5.9.0-4-amd64
/etc/kernel/postinst.d/dracut:
dracut: Generating /boot/initrd.img-5.9.0-4-amd64
mknod: /var/tmp/dracut.n115A4/initramfs/dev/kmsg: Operation not permitted
mknod: /var/tmp/dracut.n115A4/initramfs/dev/console: Operation not permitted
Setting up linux-image-amd64 (5.9.11-1) ...
Processing triggers for libc-bin (2.31-5) ...
Processing triggers for dracut (050+65-1) ...
dracut: Generating /boot/initrd.img-5.9.0-4-amd64
mknod: /var/tmp/dracut.mHlLVr/initramfs/dev/kmsg: Operation not permitted
mknod: /var/tmp/dracut.mHlLVr/initramfs/dev/console: Operation not permitted

I'm guessing enough code moved around that the comment is no longer up-to-date and now just prevents the initrd to be created.
Either way, I've tried this twice with/without the INITRD environment variable here and removing it reliably fixes the issue.

[DaanDeMeyer]

So you're not getting an initrd because unified kernel images are enabled. The initrd is baked into the efi image generated by dracut. --without-unified-kernel-images should fix the issue.

We might need to enable this by default for bios builds. I don't know if it makes much sense to use unified kernel images when doing a BIOS boot.

Thing is that for Debian, both dracut and the default initramfs generator are at play. We haven't really put in the time to sort out the initramfs story on Debian/Ubuntu unfortunately.

[whot]

I did try that, but apparently "Sorry, --without-unified-kernel-images is not supported on this distro." :)

[DaanDeMeyer]

I'll make a PR tomorrow to enable it on debian when using bios. I think I didn't enable it for debian because it breaks when using efi.