Boot failed on hosts with LVM thin/raid logical volumes

[eshikhov]

Discussed in #9140

^{Originally posted by eshikhov August 9, 2024}
Hi,

When using the lvm mirror and lvm thin pool functionality, the OS is loaded randomly.

When setting up for the first time, everything is fine, the kernel modules (dm_raid and dm_thin_pool) are loaded with the config. Then the lvm mirror and lvm thin pool volumes are created and everything works well until the server is rebooted (and the reboot can be caused by various reasons, be it talosctl upgrade with the --reboot-mode powercycle option or just talosctl reboot --mode powercycle). After a restart, talos does not start, but constantly goes into cyclic restarts and at some time of restart it can boot successfully.

Screenshots of unsuccessful launches that I managed to take:

Dump debug information about the cluster:

support.zip

Talos v1.7.6

[vladimirfx]

We reproduced the issue on VM. Logs and VMs can be found here: https://drive.google.com/drive/folders/1aj4GUOjO_EqSFdz4GgAZXhpCJ75ggZOm

In production, we are in a situation where we cannot reboot or update servers that have LV with thin or raid1 type. After reboot, the server can't boot from 30 minutes to several hours. Eventually server boots up...

Of course, all servers have the correct modules configured and LVs created in the past year (using LVM CSI).

It looks like some race in kernel module initialization.

As a workaround, can you provide a machine config option to not activate LVM at boot time? (because all known LVM CSI implementations are active LVs themselves)

[vladimirfx]

@mst1711 FYI

[smira]

Can you extract the relevant bits as text? What is the root cause that you're seeing? It's almost impossible to follow screenshots, and they are not searchable.

[vladimirfx]

Can you extract the relevant bits as text? What is the root cause that you're seeing? It's almost impossible to follow screenshots, and they are not searchable.

We provide a shared Google Disk folder with details. (https://drive.google.com/drive/folders/1aj4GUOjO_EqSFdz4GgAZXhpCJ75ggZOm)

VM boot logs from that folder:
failed-boot-serial-console-output.txt
success-boot-serial-console-output.txt

Unfortunately, we can't provide such logs from bare metal because lack of a serial console.

[eshikhov]

The problem was discovered when updating from version 1.6.7 to version 1.7.5.
I'll check it again on version 1.6 releases

[frezbo]

@eshikhov would you be able to test with a custom iso/installer?

[eshikhov]

Yes, I can try

[frezbo]

Could you sent a message on siderolabs slack?

[eshikhov]

I checked it on a custom image, the problem cannot be reproduced. After booting, LVM volumes are available. Everything works.

[vladimirfx]

@smira can the fix be backported to 1.7?

[vladimirfx]

Fantastic support, thanks!

Waiting for the release to "unlock" our servers.

[smira]

@smira can the fix be backported to 1.7?

I don't know, this seems to be bigger/breaking change if things go wrong in some setups.

[vladimirfx]

@smira can the fix be backported to 1.7?

I don't know, this seems to be bigger/breaking change if things go wrong in some setups.

I understand your concern: changing the boot sequence in the patch release is against the usual practice.
But the fixed version correctly initializes LVM (using udev rules) in contrast to unfixed which makes OS unbootable. So it is not just chore work for dropping the legacy part of the boot sequence - that effectively fixes real-user critical bugs without user-visible behavior change for others.

Anyway - thanks again for the good OS and support.

[smira]

But the fixed version correctly initializes LVM (using udev rules) in contrast to unfixed which makes OS unbootable. So it is not just chore work for dropping the legacy part of the boot sequence - that effectively fixes real-user critical bugs without user-visible behavior change for others.

yes, that's true as well, but it's too big of a change for a patch release if it breaks some setups.

Talos 1.8.0-beta.1 will be released next Monday, so you can use that.

[vladimirfx]

Talos 1.8.0-beta.1 will be released next Monday, so you can use that.

It is safe to use 1.8 worker nodes with 1.7 controll-plane?

[smira]

Talos 1.8.0-beta.1 will be released next Monday, so you can use that.

It is safe to use 1.8 worker nodes with 1.7 controll-plane?

In general no, but should be fine for this pair (1.7 <> 1.8).

[vladimirfx]

Just tested 1.8.0-beta.1 :

1. Nodes are booted normally - ok
2. Talos do not activate LVM volumes - warn

Udevd rules do not activate LVM logical volumes independent of their type (linear, thin, raid, or whatever).

For our case, it is ok, but it seems it should be nodes in release notes at least. IMO volume actuation should be done by CSI driver for their controlled volumes.

[frezbo]

@eshikhov @vladimirfx could you test with this installer image: ghcr.io/frezbo/installer:v1.8.0-alpha.2-26-g18daedb51-dirty ?

[vladimirfx]

@eshikhov @vladimirfx could you test with this installer image: ghcr.io/frezbo/installer:v1.8.0-alpha.2-26-g18daedb51-dirty ?

Of course, we will check it soon. Can you share what changes it contains?

[frezbo]

@eshikhov @vladimirfx could you test with this installer image: ghcr.io/frezbo/installer:v1.8.0-alpha.2-26-g18daedb51-dirty ?

Of course, we will check it soon. Can you share what changes it contains?

yes, restructured code to behave like a normal os, if all volume groups is available and disks backing it is available it gets activated (with events from udev). The previous implementation tried to enable every volume group unconditionally. Following this reference: https://man7.org/linux/man-pages/man7/lvmautoactivation.7.html

[vladimirfx]

yes, restructured code to behave like a normal os, if all volume groups is available and disks backing it is available it gets activated (with events from udev). The previous implementation tried to enable every volume group unconditionally. Following this reference: https://man7.org/linux/man-pages/man7/lvmautoactivation.7.html

Does it fail the boot in case of LVM activation failure?
In our case activation failed because of the kernel modules loading race (not device drivers but pure software components), so all PVs are available on boot and VGs start activating. I am scared that the new implementaion will fail the same way as the original...

[frezbo]

yes, restructured code to behave like a normal os, if all volume groups is available and disks backing it is available it gets activated (with events from udev). The previous implementation tried to enable every volume group unconditionally. Following this reference: https://man7.org/linux/man-pages/man7/lvmautoactivation.7.html

Does it fail the boot in case of LVM activation failure? In our case activation failed because of the kernel modules loading race (not device drivers but pure software components), so all PVs are available on boot and VGs start activating. I am scared that the new implementaion will fail the same way as the original...

it doesn't, it's a controller, so it'll just retry, and if lvm reports it's not healthy it just ignores

[eshikhov]

does not work on this image: ghcr.io/frezbo/installer:v1.8.0-alpha.2-31-g9fa08e843-dirty

dmesg_after_reboot.log
controller-runtime.log

[frezbo]

Hmm weird, added a test that needs dm_raid and pvscan seems to load them

[frezbo]

Are you sure these lvm volumes have auto-activation? Also would it be possible to upload support.zip from talosctl support going forward?

[eshikhov]

On the old release of lvm volumes were automatically activated

The file you requested from a non-working node:
support.zip

[vladimirfx]

I think this issue can be closed in favor of #9365 because node boot up is fixed.