RHEL 8 Amazon AMI fails on reboot

[rlmass]

Steps to reproduce behavior

Deploy any of the 2024 spel-minimal-rhel-8-hvm amis, once booted, reboot the system using "sudo reboot". Watch the serial console for the instance and it will fail to successfully start up again. (This may take atleast 2 reboots to happen)

Suggestions for fix

Looking into possible fixes, I found several guides and public forums that suggested adding "nofail" in /etc/fstab to any non-critical partitions. This seems to have fixed the issue.

https://repost.aws/questions/QUbkr4v8pBQ8Sz-4AdDVDJQA/cannot-open-access-to-console-the-root-account-is-locked-al2-t2-t3

[lorengordon]

This is a race condition in systemd version 239, distributed with EL8. Red Hat sat on the issue report and the fix for over 2 years, and now that EL8 is in maintenance mode they consider it too risky to backport the fix. We investigated in #695 (comment).

While nofail does work around the immediate boot issue, it just papers over the problem. The failure case still exists, and the system still ends up in a non-determinate state, where other things fail. It may work for your use case, but it is not a general fix for the issue.

If you have a support contract, I'd suggest opening an issue with Red Hat. Although, at this point it's probably best to upgrade to EL9 anyway.

If you do open a support case, you can reference the one we opened also about the issue, case #03863804.