Enable QEMU for eve packages build step in CI

[deitch]

When we call make pkgs eve, if a package is not available for an architecture other than the one it is running on, e.g. arm64 on an amd64 runner, the build can fail. Docker can handle emulation, but you need to enable it first. This adds enabling emulation.

This adds the standard GitHub Action to setup qemu to the eve job.

[deitch]

I have an issue with why this should be necessary. If we look at the build.yml job, it looks like the following (truncated):

jobs:
  packages:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        include:
          - os: buildjet-4vcpu-ubuntu-2204-arm
            arch: arm64
          - os: ubuntu-20.04
            arch: amd64
          - os: ubuntu-20.04
            arch: riscv64
    steps:
      - name: ensure packages for cross-arch build
        if: ${{ matrix.arch == 'riscv64' }}
        run: |
          APT_INSTALL="sudo apt install -y binfmt-support qemu-user-static"
          # the following weird statement is here to speed up the happy path
          # if the default server is responding -- we can skip apt update
          $APT_INSTALL || { sudo apt update && $APT_INSTALL ; }
      - name: Build packages
        run: |
          make V=1 PRUNE=1 ZARCH=${{ matrix.arch }} pkgs

  eve:
    needs: packages  # all packages for all platforms must be built first
    runs-on: ubuntu-20.04
    strategy:
      fail-fast: false
      matrix:
        arch: [arm64, amd64]
        hv: [xen, kvm]
        platform: ["generic"]
        include:
          - arch: riscv64
            hv: mini
            platform: "generic"
          - arch: amd64
            hv: kvm
            platform: "rt"
          - arch: arm64
            hv: kvm
            platform: "nvidia"
          - arch: arm64
            hv: kvm
            platform: "imx8mp_pollux"
          - arch: arm64
            hv: kvm
            platform: "imx8mp_epc_r3720"

    steps:
      - name: update linuxkit cache for our arch
        id: cache_for_packages
        if: ${{ matrix.arch != 'amd64' }}  # because our runner arch is amd64; if that changes, this will have to change
        uses: actions/cache/restore@v3
        with:
          path: ~/.linuxkit/cache
          key: linuxkit-${{ matrix.arch }}-${{ github.event.pull_request.head.sha }}
          fail-on-cache-miss: true

The first packages job builds the packages, each for its own arch, and does qemu setup (differently from this PR, but same result). The next eve job depends on the first, and then fills its cache from the results of the previous job.

By the time we get to make eve, we should have every package for every arch already built and in-cache.

So why is the eve job, even when it calls make eve, trying to build for other archs?

[uncleDecart]

@deitch don't we want to build EVE image for arm? I though that running it in two different stages helps us to avoid cross-compilation for arm over x64 arch

[deitch]

And yet that is exactly what is happening. Hence, yes, this should fix it, but, no, I think it shouldn't be necessary in the first place?

[uncleDecart]

Oh and I see this :DD

if: ${{ matrix.arch != 'amd64' }}  # because our runner arch is amd64; if that changes, this will have to change

[uncleDecart]

And yet that is exactly what is happening. Hence, yes, this should fix it, but, no, I think it shouldn't be necessary in the first place?

I remember adding platform parameter for our build.yml which enabled us to publish artifacts of EVE compiled for NVIDIA board, for instance and that @rene wanted it

My assumption was also that pkgs were heavy-weighted things, meaning we are using cache for natively building packages on arm and then in the next step we're just bringing it all together

[deitch]

Oh and I see this

Look at the multi-line comment a few steps above. That was necessary because some of the things we use are for including in the image (so depends on our target OS), and some are for building the image (so depends on our builder OS). So we load builder OS stuff first, get it in place, then load target OS stuff.

My assumption was also that pkgs were heavy-weighted things, meaning we are using cache for natively building packages on arm and then in the next step we're just bringing it all together

Mine, too. Which is why I am asking, why is this building pkgs again at that stage?

[uncleDecart]

My assumption was also that pkgs were heavy-weighted things, meaning we are using cache for natively building packages on arm and then in the next step we're just bringing it all together

Mine, too. Which is why I am asking, why is this building pkgs again at that stage?

Huh, I thought that our makefile would pick up cache and just build for the specific arch it was targeted, that's why we have matrix for it...

[deitch]

Aha! See this: https://github.com/lf-edge/eve/actions/runs/11009931353/job/30571118821?pr=4277#step:7:9

[deitch]

That should not have failed silently.

[deitch]

It is the right cache key, too, I checked. linuxkit-arm64-8783a274d9e2924f1e526d80c537fecec1eb289b

[deitch]

I switched this to a draft. Actually, I never want this merged in.

If a package was not built by the time the eve job gets triggered, we want it to fail. This is a sign that something was broken earlier.

I will close this once the discussion is done.

[uncleDecart]

Could it be different layout for filesystem, I wonder what buildjet is using for their arm runner. Although they should be compatible with ext4 I guess...

[uncleDecart]

If a package was not built by the time the eve job gets triggered, we want it to fail. This is a sign that something was broken earlier.

I guess this could be done with if statement, which will skip the next step, but I'm not sure that this is the behaviour we want...

[deitch]

Cache creation:

/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/actions-runner/_work/eve/eve --files-from manifest.txt --use-compress-program zstdmt

Cache restore:

/usr/bin/tar -xf /home/runner/work/_temp/c633f3b4-dfd0-489b-9503-993b4590dd63/cache.tzst -P -C /home/runner/work/eve/eve --use-compress-program unzstd

/usr/bin/tar: ../../../../.linuxkit: Cannot mkdir: Permission denied
/usr/bin/tar: ../../../../.linuxkit/cache: Cannot mkdir: No such file or directory

And @uncleDecart wins the prize! Why they are doing those weird paths is beyond me, but they are, and that is the issue. Now I know how to track it down.

[deitch]

Closing this in favour of #4287