Request for info: running hermit-based applications on top of Xen

[lyuts]

Hi guys,

Thank you for this project and your effort, this looks very interesting and promising. Out of curiosity I decided to give a try of hello world in one of test xen installations.

Compiled with

cargo build -Zbuild-std=std,core,alloc,panic_abort -Zbuild-std-features=compiler-builtins-mem --target x86_64-unknown-hermit

I grabbed the produced binary, created an empty initramfs and tried spinning up a domU. Guest failed to run, and I'm getting the following errors in the xl logs.

2022-12-27 02:10:11.988+0000: xc: panic: xc_dom_hvmloader.c:112: xc_dom_parse_hvm_kernel: ELF image is not 32bit: Invalid kernel
2022-12-27 02:10:11.988+0000: libxl: libxl_dom.c:578:libxl__build_dom: xc_dom_parse_image failed
2022-12-27 02:10:11.988+0000: libxl: libxl_create.c:1654:domcreate_rebuild_done: Domain 53:cannot (re-)build domain: -3

$ file target/x86_64-unknown-hermit/debug/unikernel_demo
target/x86_64-unknown-hermit/debug/unikernel_demo: ELF 64-bit LSB pie executable, x86-64, version 1 (embedded), dynamically linked, BuildID[xxHash]=3122861d032a21a2, with debug_info, not stripped

This is probably not a bug, and instead I'm looking for pointers in investigating this or proper ways to run this on Xen. Any bits of information would be useful.
Thank you.

[lyuts]

Update: I should confirm that I'm on a 64bit host, and 32 bit error message looks like a red herring. I realized that I'm just feeding in a produced binary as is. I compressed it and I'm now getting a more descriptive error:

xc: panic: xc_dom_core.c:692: xc_dom_find_loader: no loader found: Invalid kernel

which makes sense, and I need to figure out how to bundle both loader and kernel when creating the guest.

[stlankes]

Excuse me, I oversaw this issue. I know this issue. Currently, we don't have the time to support Xen.

[lyuts]

@stlankes , do you think would it be possible for you and/or other project core team to document Xen support, i.e. issues, blockers, and what bits are missing? I myself would find it very useful for studying the domain, and maybe someone from the community could step up and help with the implementation.

[hydroid7]

Here is a blog post about hermit core. It may gives you some hints:
https://janma.tk/2018-11-11/os-xen-001/

And here is the source code:
https://gitlab.com/JanMa/HermitCore/tree/xen_hvm

[stlankes]

Thank you @hydroid7. I forgot this issue. @lyuts Are you still interested? I think that we can do it.

[lyuts]

@hydroid7 thank you for the useful links. @stlankes, yes I still have a use case for this.

[stlankes]

Ok, I will check it. But not this year :-)

[lyuts]

@stlankes, how would you guys typically tackle this kind of changes/project, one big pull request with complete support, or incrementally adding support through many piecemeal changes?
Happy holidays!

[jtracey]

I played around a bit with the blog's instructions for building and running an image as a hvm, and got "hello world" working. Specifically, I created the folder structure described for creating a GRUB ISO, replacing hermit_application with the hermit application and ldhermit.elf with the hermit loader (including the -uart=io:0x3f8 argument), then ran grub-mkrescue -o /tmp/hermit.iso iso/. I then created a xen domain configuration for it:

type = "hvm"
name = "hermit-single.hvm"
memory = 1024
vcpus = 1
disk = [ 'file:/tmp/hermit.iso,hdc:cdrom,r' ]
nographic = 1
serial = [ 'file:/tmp/hermit.log' ]

It launched and logged to the file when I ran it (via xl create), but the hermit application crashed:

Crash log

[LOADER][INFO] Loader: [0x100000 - 0x121059]
[LOADER][INFO] Found Multiboot information at 0x10000
[LOADER][WARN] Mapping 1 4KiB pages from 0x10000..0x11000 to 0x10000..0x11000
[LOADER][INFO] Found an ELF module at [0x125000 - 0x1875458]
[LOADER][INFO] Module length: 0x1750458
[LOADER][WARN] Mapping 4KiB pages starting from 0x125000 to frames 0x125000..0x200000
[LOADER][WARN] Mapping 2MiB pages starting from 0x200000 to frames 0x200000..0x1a00000
[LOADER][INFO] Parsing kernel from ELF at 0x125000..0x1875458 (len = 0x1750458 B / 24446040 B)
[LOADER][WARN] Mapping 3 2MiB pages from 0x1a00000..0x2000000 to 0x1a00000..0x2000000
[LOADER][INFO] Loading kernel to 0x1a00000..0x1f82cf8 (len = 0x582cf8 B / 5778680 B)
[LOADER][INFO] TLS is at 0x1e37718..0x1e37788 (len =  0x70 B / 112 B)
[LOADER][WARN] Mapping 8 4KiB pages from 0x11000..0x19000 to 0x11000..0x19000
[LOADER][INFO] boot_info = BootInfo {
    hardware_info: HardwareInfo {
        phys_addr_range: 0x0..0x0,
        serial_port_base: Some(
            0x3f8,
        ),
        device_tree: Some(
            0x12053b,
        ),
    },
    load_info: LoadInfo {
        kernel_image_addr_range: 0x1a00000..0x1f82cf8,
        tls_info: Some(
            TlsInfo {
                start: 0x1e37718,
                filesz: 0x20,
                memsz: 0x70,
                align: 0x8,
            },
        ),
    },
    platform_info: Multiboot {
        command_line: Some(
            "",
        ),
        multiboot_info_addr: 0x10000,
    },
}
[LOADER][INFO] Entering kernel at 0x1d23040, stack at 0x11000, raw_boot_info at 0x11f008
[0][INFO] Welcome to Hermit 0.8.0
[0][INFO] Kernel starts at 1a00000
[0][INFO] BSS starts at 0x1f81cf8
[0][INFO] tls_info = Some(
    TlsInfo {
        start: 0x1e37718,
        filesz: 0x20,
        memsz: 0x70,
        align: 0x8,
    },
)
[0][INFO] Total memory size: 983 MB
[0][INFO] Kernel region: [1a00000 - 2000000]
[0][INFO] A pure Rust application is running on top of Hermit!
[0][INFO] Heap: size 850 MB, start address 2000000
[0][INFO] Heap is located at 0x2000000..0x37200000 (0 Bytes unmapped)
[0][INFO] Physical memory free list:
        0x37200000..        0x3f7ff000 (len =          0x85ff000, pages =            34303)
[0][INFO] Virtual memory free list:
        0x37200000..    0x800000000000 (len =     0x7fffc8e00000, pages =      34359512576)
[0][INFO] 
[0][INFO] ========================== CPU INFORMATION ===========================
[0][INFO] Model:                   13th Gen Intel(R) Core(TM) i5-1340P
[0][INFO] Frequency:               2188 MHz (from Measurement)
[0][INFO] SpeedStep Technology:    Not Available
[0][INFO] Features:                MMX SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AESNI RDRAND FMA MOVBE MCE FXSR XSAVE RDTSCP CLFLUSH TSC-DEADLINE X2APIC HYPERVISOR AVX2 BMI1 BMI2 PKU FSGSBASE RDSEED 
[0][INFO] Physical Address Width:  39 bits
[0][INFO] Linear Address Width:    48 bits
[0][INFO] Supports 1GiB Pages:     Yes
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Hermit booted on 2024-12-31 20:43:23.064697 +00:00:00
[0][INFO] Found an ACPI revision 2 table at 0xF5B50 with OEM ID "Xen"
[0][INFO] IOAPIC v17 has 48 entries
[0][INFO] Disable IOAPIC timer
[0][ERROR] IOAPIC: trying to turn on irq 25 which is too high
[0][PANIC] panicked at src/arch/x86_64/kernel/apic.rs:531:72:

called Result::unwrap() on an Err value: ()

Number of interrupts

[0][INFO] shutting down with code 1

This appears to be because the Hermit kernel doesn't support multiple I/O APICs, only allowing up to 24 IRQs (the main branch is slightly different, but effectively the same). I hacked around it by only looping over the first 24, which was enough to get the whole program to succeed:

Successful log

[LOADER][INFO] Loader: [0x100000 - 0x121059]
[LOADER][INFO] Found Multiboot information at 0x10000
[LOADER][WARN] Mapping 1 4KiB pages from 0x10000..0x11000 to 0x10000..0x11000
[LOADER][INFO] Found an ELF module at [0x125000 - 0x1876150]
[LOADER][INFO] Module length: 0x1751150
[LOADER][WARN] Mapping 4KiB pages starting from 0x125000 to frames 0x125000..0x200000
[LOADER][WARN] Mapping 2MiB pages starting from 0x200000 to frames 0x200000..0x1a00000
[LOADER][INFO] Parsing kernel from ELF at 0x125000..0x1876150 (len = 0x1751150 B / 24449360 B)
[LOADER][WARN] Mapping 3 2MiB pages from 0x1a00000..0x2000000 to 0x1a00000..0x2000000
[LOADER][INFO] Loading kernel to 0x1a00000..0x1f82e48 (len = 0x582e48 B / 5779016 B)
[LOADER][INFO] TLS is at 0x1e37858..0x1e378c8 (len =  0x70 B / 112 B)
[LOADER][WARN] Mapping 8 4KiB pages from 0x11000..0x19000 to 0x11000..0x19000
[LOADER][INFO] boot_info = BootInfo {
    hardware_info: HardwareInfo {
        phys_addr_range: 0x0..0x0,
        serial_port_base: Some(
            0x3f8,
        ),
        device_tree: Some(
            0x12053b,
        ),
    },
    load_info: LoadInfo {
        kernel_image_addr_range: 0x1a00000..0x1f82e48,
        tls_info: Some(
            TlsInfo {
                start: 0x1e37858,
                filesz: 0x20,
                memsz: 0x70,
                align: 0x8,
            },
        ),
    },
    platform_info: Multiboot {
        command_line: Some(
            "",
        ),
        multiboot_info_addr: 0x10000,
    },
}
[LOADER][INFO] Entering kernel at 0x1cf7130, stack at 0x11000, raw_boot_info at 0x11f008
[0][INFO] Welcome to Hermit 0.8.0
[0][INFO] Kernel starts at 1a00000
[0][INFO] BSS starts at 0x1f81e58
[0][INFO] tls_info = Some(
    TlsInfo {
        start: 0x1e37858,
        filesz: 0x20,
        memsz: 0x70,
        align: 0x8,
    },
)
[0][INFO] Total memory size: 983 MB
[0][INFO] Kernel region: [1a00000 - 2000000]
[0][INFO] A pure Rust application is running on top of Hermit!
[0][INFO] Heap: size 850 MB, start address 2000000
[0][INFO] Heap is located at 0x2000000..0x37200000 (0 Bytes unmapped)
[0][INFO] Physical memory free list:
        0x37200000..        0x3f7ff000 (len =          0x85ff000, pages =            34303)
[0][INFO] Virtual memory free list:
        0x37200000..    0x800000000000 (len =     0x7fffc8e00000, pages =      34359512576)
[0][INFO] 
[0][INFO] ========================== CPU INFORMATION ===========================
[0][INFO] Model:                   13th Gen Intel(R) Core(TM) i5-1340P
[0][INFO] Frequency:               2189 MHz (from Measurement)
[0][INFO] SpeedStep Technology:    Not Available
[0][INFO] Features:                MMX SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 AVX AESNI RDRAND FMA MOVBE MCE FXSR XSAVE RDTSCP CLFLUSH TSC-DEADLINE X2APIC HYPERVISOR AVX2 BMI1 BMI2 PKU FSGSBASE RDSEED 
[0][INFO] Physical Address Width:  39 bits
[0][INFO] Linear Address Width:    48 bits
[0][INFO] Supports 1GiB Pages:     Yes
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Hermit booted on 2024-12-31 22:10:13.379534 +00:00:00
[0][INFO] Found an ACPI revision 2 table at 0xF5B50 with OEM ID "Xen"
[0][INFO] IOAPIC v17 has 48 entries
[0][INFO] Disable IOAPIC timer
[0][INFO] 
[0][INFO] ===================== MULTIPROCESSOR INFORMATION =====================
[0][INFO] APIC in use:             x2APIC
[0][INFO] Initialized CPUs:        1
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Compiled with PCI support
[0][INFO] Compiled with ACPI support
[0][INFO] Compiled with FSGSBASE support
[0][INFO] Compiled with SMP support
[0][INFO] 
[0][INFO] ======================== PCI BUS INFORMATION =========================
[0][INFO] 00:00 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237]
[0][INFO] 00:01 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
[0][INFO] 00:02 Unassigned class [FF80]: XenSource, Inc. Xen Platform Device [5853:0001], IRQ 11, BAR0 IO { port: 0xC000 }, BAR1 Memory32 { address: 0xF2000000, size: 0x1000000, prefetchable: true }
[0][INFO] 00:03 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00B8], BAR0 Memory32 { address: 0xF0000000, size: 0x2000000, prefetchable: true }, BAR1 Memory32 { address: 0xF3010000, size: 0x1000, prefetchable: false }
[0][INFO] ======================================================================
[0][INFO] 
[0][INFO] Hermit is running on common system!
[0][INFO] Try to initialize network!
[0][INFO] Try to initialize uhyve filesystem
Hello, world!
Number of interrupts
[0][FPU]: 1
[0][INFO] shutting down with code 0