linux-5.4.88-modules (aarch64) fails, blocking nixos-unstable

[FRidh]

Describe the bug
https://hydra.nixos.org/build/135054936

Caused by #78430.

To Reproduce
Steps to reproduce the behavior:

1. ...
2. ...
3. ...

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Notify maintainers

cc @NeQuissimus

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

[primeos]

The error (full log):

root module: vmw_balloon
modprobe: FATAL: Module vmw_balloon not found in directory /nix/store/877vj59c34mrwa7b6120hdf02rkb5y08-linux-5.4.88-modules/lib/modules/5.4.88
builder for '/nix/store/1yhfy63iaxg1adlzwp91bxf6076863ij-linux-5.4.88-modules-shrunk.drv' failed with exit code 1

Seems to be related to nixos/modules/profiles/all-hardware.nix (vmw_balloon was added in 870375e):

      # VMware support.
      "mptspi" "vmw_balloon" "vmwgfx" "vmw_vmci" "vmw_vsock_vmci_transport" "vmxnet3" "vsock"

Not sure why the vmw_balloon kernel module seems to be suddenly gone.

cc @volth (disabled by GitHub...)

Update: ${linux}/lib/modules/5.4.88/kernel/drivers/misc/vmw_balloon.ko.xz is still there so this seems to be something else (modprobe issue? might be something completely unrelated or a transient failure).

[primeos]

Update: Ok, so I got a bit confused by the Hydra UI as e.g. https://hydra.nixos.org/build/135057055 states "System: x86_64-linux" but the failing build is from an aarch64-linux machine (e2da2215). I guess this is because we don't have a dedicated channel for AArch64.

In any case as long as the failure only occurs on AArch64 builds that would explain it (the vmw_balloon module depends on x86):

config VMWARE_BALLOON
        tristate "VMware Balloon Driver"
        depends on VMWARE_VMCI && X86 && HYPERVISOR_GUEST
        select MEMORY_BALLOON
        help
          This is VMware physical memory management driver which acts
          like a "balloon" that can be inflated to reclaim physical pages
          by reserving them in the guest and invalidating them in the
          monitor, freeing up the underlying machine pages so they can
          be allocated to other guests. The balloon can also be deflated
          to allow the guest to use more physical memory.

          If unsure, say N.

          To compile this driver as a module, choose M here: the
          module will be called vmw_balloon.

But I guess the questions still is what changed in Nixpkgs to cause this regression.

Edit: Seems like this does indeed only affect nixos.iso_minimal.aarch64-linux (x86 is fine: iso_minimal.x86_64-linux). The strange thing is that this error occured before but was ignored:

https://hydra.nixos.org/build/134729815 (log):

[...]
root module: virtio_balloon
  builtin dependency: virtio_balloon
root module: virtio_console
  builtin dependency: virtio_console
root module: mptspi
  dependency already copied: /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88/kernel/drivers/scsi/scsi_transport_spi.ko.xz
  copying dependency: /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88/kernel/drivers/message/fusion/mptbase.ko.xz
  copying dependency: /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88/kernel/drivers/message/fusion/mptscsih.ko.xz
  copying dependency: /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88/kernel/drivers/message/fusion/mptspi.ko.xz
root module: vmw_balloon
modprobe: FATAL: Module vmw_balloon not found in directory /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88
  not found
root module: vmwgfx
modprobe: FATAL: Module vmwgfx not found in directory /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88
  not found
root module: vmw_vmci
modprobe: FATAL: Module vmw_vmci not found in directory /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88
  not found
root module: vmw_vsock_vmci_transport
modprobe: FATAL: Module vmw_vsock_vmci_transport not found in directory /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88
  not found
root module: vmxnet3
  copying dependency: /nix/store/gwna9s2xs4x6l7hy9i9w8prbz0wl7763-linux-5.4.88-modules/lib/modules/5.4.88/kernel/drivers/net/vmxnet3/vmxnet3.ko.xz
[...]

[FRidh]

maybe just the kernel update? a9dfa24

[primeos]

Think I finally found it: fdf3215

cc @puckipedia

[primeos]

So I guess it would be best to revert that commit in master for now and try to reland it via staging/staging-next as the log contains multiple modules that are "not found" (10 matches) and this might also affect other things (maybe additional builds/tests but it could also cause regressions outside of Nixpkgs as this changes the behaviour). But we could of course also try to fix the errors in master instead if someone prefers that.

[vcunat]

I already have code that adds all those conditionals to make the minimal ISO build, so we could use that if that's the only issue.

[FRidh]

I already have code that adds all those conditionals to make the minimal ISO build, so we could use that if that's the only issue.

suggest we try and see

[vcunat]

Ok, so I got a bit confused by the Hydra UI as e.g. https://hydra.nixos.org/build/135057055 states "System: x86_64-linux" but the failing build is from an aarch64-linux machine (e2da2215). I guess this is because we don't have a dedicated channel for AArch64.

You're looking at the tested meta-job that combines things from multiple platforms.

[primeos]

You're looking at the tested meta-job that combines things from multiple platforms.

Yeah, I didn't know that it combines things from multiple platforms this way (thought there would be two tested jobsets) but I guess that makes sense in hindsight. But in any case it would IMO be nice to have dedicated channels for each architecture (cc #83049) but that is another topic (and I didn't really consider the details).

Anyway, thanks for pushing a fix :)

[vcunat]

:-) The trunk-combined jobset bears lots of history, just look at the name from svn times...