udev gets stuck after reload not initializing devices

[lorenz]

Describe the bug

systemd-udev doesn't initialize all devices after a reload. This causes various follow-on issues like #180175.

Steps To Reproduce

Steps to reproduce the behavior:

1. Switch to a different version of udev (or just a rebuild)
2. sudo udevadm info /sys/class/net/wg0

The device is not initialized (note the lack of USEC_INITIALIZED):

P: /devices/virtual/net/wg0
L: 0
E: DEVPATH=/devices/virtual/net/wg0
E: DEVTYPE=wireguard
E: INTERFACE=wg0
E: IFINDEX=6
E: SUBSYSTEM=net

Expected behavior

sudo udevadm info /sys/class/net/wg0
P: /devices/virtual/net/wg0
L: 0
E: DEVPATH=/devices/virtual/net/wg0
E: DEVTYPE=wireguard
E: INTERFACE=wg0
E: IFINDEX=6
E: SUBSYSTEM=net
E: USEC_INITIALIZED=5650399976
E: PATH=/nix/store/41pz7bsljf9nnj154vzrvand881ij31c-udev-path/bin:/nix/store/41pz7bsljf9nnj154vzrvand881ij31c-udev-path/sbin
E: ID_MM_CANDIDATE=1
E: ID_NET_LINK_FILE=/nix/store/iirqa1lqni18mcaii3g1l5s15jxwz17a-systemd-250.4/lib/systemd/network/99-default.link
E: ID_NET_NAME=wg0
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/wg0
E: TAGS=:systemd:
E: CURRENT_TAGS=:systemd:

Additional context

Probably an upstream bug triggered by NixOS because it likes to reload udev.

Notify maintainers

@flokli @kloenk @Mic92

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.52, NixOS, 22.05 (Quokka), 22.05.1859.c93e5ab157b`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.8.1`
 - channels(root): `"nixos-22.05, nixpkgs-22.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

[neilmayhew]

In my case, it's /sys/class/net/vboxnet0 and not /sys/class/net/wg0. It's been happening consistently for over a year.

The workaround in #195777 (comment) fixes the lack of initialization. However, the value of USEC_INITIALIZED is quite high, so it looks like the kernel thinks the device has been initialized all along.

[AkechiShiro]

Is there any fix, we can work towards, should we change the parameter USEC_INITIALIZED ?

[lorenz]

The lack of USEC_INITIALIZED is just indicative of an issue in udev. udev sets this parameter once it has initialized a device. But this is not happening here, at least for some devices after an udev restart.

[AkechiShiro]

So I take it we should open an issue upstream and discuss a solution with udev devs ?

[lorenz]

Yeah, this is most likely an upstream issue. I put systemd.services.systemd-udevd.restartIfChanged = false; which worked around the issue for me as not restarting udev means it doesn't need to reinitialize devices.

[AkechiShiro]

I've opened an issue upstream : systemd/systemd#34585

Hopefully we can get to the bottom of this issue