install: rpi3/4 targets

[blaggacao]

I've been biting through with some inspiration taken from the ./installer.sh
My proposal would be to incorporate the relevant bits into the installer to enable a RPi3/4 installation. Please give feedback on the design in order to check if it's worth weaving this into the installer.sh.

If so, it would require a --rpi4 / --rpi3 flag, since it branches quite a bit on the preparation of the esp partition. All other boards, that shall be eventually supported could be implemented through a different branch for preparing this esp partition.

Using EFISTUB

I wonder if it's necesary on behalf of k3os to run with grub, or if, instead we could use https://www.kernel.org/doc/html/latest/admin-guide/efi-stub.html

Looks like at least, focal fossa, has it enabled: https://kernel.ubuntu.com/git/ubuntu/ubuntu-focal.git/tree/debian.master/config/config.common.ubuntu#n3041

On the other hand, that would forgo the benefits of squashfs.

grup prefix from the host Problem

# --bootloader-id=debian is a hack for https://github.com/rhboot/grub2/blob/2a2e10c1b39672de3d5da037a50d5c371f49b40d/configure.ac#L305
# since we are using the debian packaged binaries

It would be nice, to have this prefix properly set to k3os, instead. It might be even a bug of grub-install to not respect --bootloader-id=k3os when bundling the prefix into the efi image. Maybe the only viable alternative is using grub-mkimage or grub-mkstandalone to work around this.

Rpi 3 / 4 for k3os

1. create two boot partitions:

   • K3OS_ESP (50 MB)
   • K3OS_STATE (rest)

2. Unpack filesystems:

   • k3os rootfs onto K3OS_STATE
   • rpi-uefi-fs onto K3OS_ESP

3. Downaload grub-efi from debian:

   • grub-efi-arm64
   • grub-efi-arm64-signed

4. Grub install with efi to K3OS_ESP

DEVICE=/dev/mmcblk0

K3OS_ROOTFS_URL=https://github.com/rancher/k3os/releases/download/v0.10.1/k3os-rootfs-arm64.tar.gz
RPI3_UEFI_URL=https://github.com/pftf/RPi3/releases/download/v1.25/RPi3_UEFI_Firmware_v1.25.zip
RPI4_UEFI_URL=https://github.com/pftf/RPi4/releases/download/v1.13/RPi4_UEFI_Firmware_v1.13.zip

GRUB_ARM64_DEB_URL=http://ftp.debian.org/debian/pool/main/g/grub2/grub-efi-arm64-bin_2.04-7_arm64.deb
GRUB_ARM64_SIGNED_DEB_URL=http://ftp.debian.org/debian/pool/main/g/grub-efi-arm64-signed/grub-efi-arm64-signed_1+2.04+7_arm64.deb

get_url()
{
    FROM=$1
    TO=$2
    case $FROM in
        ftp*|http*|tftp*)
            curl -o $TO -fL ${FROM}
            ;;
        *)
            cp -f $FROM $TO
            ;;
    esac
}


dd if=/dev/zero of=${DEVICE} bs=1M count=1

# https://github.com/pftf/RPi3
# Note: Do not try to use GPT for the partition scheme or 0xef (EFI System
# Partition) for the type, as these are unsupported by the CPU-embedded
# bootloader. (of the raspberry pi)
/usr/sbin/parted -s ${DEVICE} mklabel msdos  # mbr
/usr/sbin/parted -s ${DEVICE} mkpart primary fat16 0% 50MB
/usr/sbin/parted -s ${DEVICE} mkpart primary ext4 50MB 100%
/usr/sbin/parted -s ${DEVICE} set 1 boot on  # esp -> efi system partition type, see above
/usr/sbin/partprobe ${DEVICE} 2>/dev/null || true
BOOT=${DEVICE}p1
STATE=${DEVICE}p2
/usr/sbin/mkfs.vfat -F 16 ${BOOT}
/usr/sbin/fatlabel ${BOOT} K3OS_ESP
/usr/sbin/mkfs.ext4 -F -L K3OS_STATE ${STATE}

TARGET=/run/k3os/target
mkdir -p ${TARGET}
mount ${STATE} ${TARGET}
mkdir -p ${TARGET}/boot/efi
mkdir -p ${TARGET}/boot/grup
mount ${BOOT} ${TARGET}/boot/efi


# Unpack k3os root filesystem to ${TARGET}
TEMP_FILE=$(mktemp k3os.XXXXXXXX.tar.gz)
get_url ${K3OS_ROOTFS_URL} ${TEMP_FILE}
tar zxvf ${TEMP_FILE} --strip-components=1 -C ${TARGET}
rm -f $TEMP_FILE

# Unpack Rpi UEFI to ${TARGET}/boot/efi
TEMP_FILE=$(mktemp rpiuefi.XXXXXXXX.zip)
get_url ${RPI3_UEFI_URL} ${TEMP_FILE}
unzip ${TEMP_FILE} -d ${TARGET}/boot/efi
rm -f $TEMP_FILE

# Unpack GRUB_ARM64_DEB_URL & GRUB_ARM64_SIGNED_DEB_URL
TEMP_FILE1=$(mktemp grubarm64.XXXXXXXX.deb)
TEMP_FILE2=$(mktemp grubarm64-signed.XXXXXXXX.deb)
TEMP_DIR=$(mktemp -d grubarm64.XXXXXXXX)
get_url ${GRUB_ARM64_DEB_URL} ${TEMP_FILE1}
ar x ${TEMP_FILE1} --output=${TEMP_DIR} data.tar.xz
tar xvf ${TEMP_DIR}/data.tar.xz --strip-components=1 -C ${TEMP_DIR}
rm -f ${TEMP_DIR}/data.tar.xz

get_url ${GRUB_ARM64_SIGNED_DEB_URL} ${TEMP_FILE2}
ar x ${TEMP_FILE2} --output=${TEMP_DIR} data.tar.xz
tar xvf ${TEMP_DIR}/data.tar.xz --strip-components=1 -C ${TEMP_DIR}
rm -f ${TEMP_DIR}/data.tar.xz

cat > ${TARGET}/boot/grub/grub.cfg << EOF
set default=0
set timeout=10
set gfxmode=auto
set gfxpayload=keep
insmod all_video
insmod gfxterm
menuentry "k3OS Current" {
  search.fs_label K3OS_STATE root
  set sqfile=/k3os/system/kernel/current/kernel.squashfs
  loopback loop0 /\$sqfile
  set root=(\$root)
  linux (loop0)/vmlinuz printk.devkmsg=on console=tty1 $GRUB_DEBUG
  initrd /k3os/system/kernel/current/initrd
}
menuentry "k3OS Previous" {
  search.fs_label K3OS_STATE root
  set root=(\$root)
  linux /k3os/system/kernel/previous/vmlinuz printk.devkmsg=on console=tty1 $GRUB_DEBUG
  initrd /k3os/system/kernel/previous/initrd
}
menuentry "k3OS Rescue Shell" {
  search.fs_label K3OS_STATE root
  set root=(\$root)
  linux /k3os/system/kernel/current/vmlinuz printk.devkmsg=on rescue console=tty1
  initrd /k3os/system/kernel/current/initrd
}
EOF

# --bootloader-id=debian is a hack for https://github.com/rhboot/grub2/blob/2a2e10c1b39672de3d5da037a50d5c371f49b40d/configure.ac#L305
# since we are using the debian packaged binaries
grub-install --directory=${TEMP_DIR}/usr/lib/grub/arm64-efi --boot-directory=${TARGET}/boot --efi-directory=${TARGET}/boot/efi --uefi-secure-boot --bootloader-id=debian

rm -f $TEMP_FILE1
rm -f $TEMP_FILE2
rm -rf $TEMP_DIR

umount ${TARGET}/boot/efi || true
umount ${TARGET} || true

[blaggacao]

closes #309 #111 #399 presumably #345, towards #65

[blaggacao]

@sgielen would you see a vested disadvantage using the uefi firmware over raspberry foundation firmware?

This is it's default config.txt

arm_64bit=1
disable_commandline_tags=2
disable_overscan=1
enable_uart=1
uart_2ndstage=1
armstub=RPI_EFI.fd
device_tree_address=0x1f0000
device_tree_end=0x200000

devicetrees themselves are taken from upstream and probably are not too volatile.

[sgielen]

@sgielen would you see a vested disadvantage using the uefi firmware over raspberry foundation firmware?

I actually have no experience with using firmware other than the default. To be clear, are you talking about the firmware flashed onto the Pi's ROM or the firmware residing on the boot partition of the SD card?

A pull request by @brian-provenzano (sgielen/picl-k3os-image-generator#15) was merged today that allows defining what Raspberry Pi firmware version to use for the SD card boot partition; perhaps the code could be amended to choose other firmware sources as well.

[blaggacao]

Thanks for letting me know. I was referring to the firmware that gets placed on the SD card, indeed.

I'd hope to get the discussion going to get it upstreamed.

[rharmonson]

Any impact if booting using USB? Raspberry PI 4's eeprom supports USB boot now.

https://www.raspberrypi.org/documentation/hardware/raspberrypi/booteeprom.md
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

[frudolph77]

Hello @blaggacao

I'm trying to use your script to get k3os running on a RPI3.
I've updated to the latest UEFI-Firmware 1.32, and the latest grub version 2.04-15, the -7 is no longer available.

There is at least one typo inside grup instead of grub and I also removed the /usr/sbin paths since the binaries are in a different location on my Ubuntu System.

Finally I managed to burn the SD, but got an error grub-install: Fehler: Neuzuweisung 0x113 ist noch nicht implementiert. I just ignored it and tried the SD in my PI

It boots up, I can enter the shell and the Firmware menu, but it wont boot k3os. I get this error:

BdsDex: No bootable option or device was found
BdsDex: Press any key to enter the Boot Manager Menu.

Kind regards
Frank

[frezbo]

@blaggacao this is awesome 👍 Did you tested this on a RPi4? as per the uefi firmware the default RAM is limited to 3Gigs. I really like this solution since it uses the kernel packaged with k3os

[blaggacao]

@frezbo Out of convenience, I have shifted away from k3os in favor of NixOS (combined with deploy-rs) for host os deployments.

I'm in the adoption of divnix/devos framework in managing my whole environment (k3s nodes and "clasic" hosts). That includes initial bespoke sd card burning of rpi3/rpi4 hosts, which hopefully will make it soon into devos.

Sure enough k3os could be packages as nix package and be deployed as nix module in a similar way, based on my inital work above and based on the referenced work in the nix ecosystem.

[frezbo]

@blaggacao That seems interesting, I also checked out the mentioned links and it seems nix is using u-boot, is that a fair assumption? I'm not much familiar with Nix, do you have anything that can help me get started

[blaggacao]

@frezbo Nix is three things:

• an extremely powerful configuration language (let's call it nix-lang). Similar to dhall, cue-lang, or jsonnet
• a cli nix
• an OS distribution (NixOS)

NixOS/nixpkgs is the biggest and most up-to-date build instructions database (source: Repology), but also home to NixOS configuration.

To get started with a NixOS based environment I highly recommend forking the core or community branch of divnix/devos, then adapt and merge upstream divnix/devos regularily as it evolves fastly.

The learning curve is a bit stretched, in general, but the rewards are nothing short of a devops dream.

NixOS can use u-boot, and as far as I eemember does by default for aarch64/arm7 based boards.

https://github.com/nix-community/nixos-generators might be a good tool for easily generating images that NixOS stock (or divnix/devos) does not yet provide already.

Reading through this thread again, nothing stops us from proposinh to build and deploy k3os through nix-lang and nixpkgs' powerful configuration module system. That would be quite interesting and my initial command history that I posted here might actually be reused for that.