Bazzite not updating grub after upgrade to 41 (stuck in 39)

[kikito]

Describe the bug

My system seems to be stuck in Bazzite 39, I can't seem to be able to upgrade to 41. I suspect a problem updating the grub menu.

What did you expect to happen?

I expected the Bazzite 41 option to appear on the Grub2 boot menu after updating, but it has not.

• Discover says I am "Up to date"
• I have also followed the upgrading guide. For good measure, I just executed ujust update on the console right now.

Output of `ujust upgrade`

── 10:40:30 - System update ────────────────────────────────────────────────────
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
Checking out tree 5593736... done
No upgrade available.

── 10:40:37 - Distrobox ────────────────────────────────────────────────────────

── 10:40:37 - Firmware upgrades ────────────────────────────────────────────────
Metadata is up to date; use --force to refresh again.
Devices with no available firmware updates: 
 • BCM20702A0
 • System Firmware
 • UEFI Device Firmware
 • UEFI Device Firmware
 • WD BLACK SN770 1TB
Devices with the latest available firmware version:
 • UEFI dbx
No updates available

── 10:40:37 - Flatpak User Packages ────────────────────────────────────────────
Looking for updates…

Info: runtime org.kde.Platform branch 6.6 is end-of-life, with reason:
   We strongly recommend moving to the latest stable version of the Platform and SDK
Info: applications using this runtime:
   com.obsproject.Studio

Nothing to do.

── 10:40:38 - Flatpak System Packages ──────────────────────────────────────────
Looking for updates…

Info: app org.ryujinx.Ryujinx branch stable is end-of-life, with reason:
   This application is no longer maintained.

Nothing to do.

── 10:40:38 - Nix ──────────────────────────────────────────────────────────────
unpacking channels...

── 10:40:38 - pip3 ─────────────────────────────────────────────────────────────
Requirement already satisfied: pip in /home/kikito/.local/lib/python3.12/site-packages (24.3.1)

── 10:40:38 - Linux AMDGPU Control Application ─────────────────────────────────
Looking for updates...
LACT not installed, skipping.

── 10:40:39 - Mozilla GNOME Themes ─────────────────────────────────────────────
Looking for updates...

── 10:40:39 - Third Party CSS Loader Themes ────────────────────────────────────
Looking for updates...

── 10:40:39 - Summary ──────────────────────────────────────────────────────────
System update: OK
distrobox: OK
Firmware upgrades: OK
Flatpak: OK
nix: OK
pip3: OK
Linux AMDGPU Control Application: OK
Mozilla GNOME Themes: OK
Third Party CSS Loader Themes: OK

(R)eboot
(S)hell
(Q)uit

Pressing "quit" I get the message "topgrade finished successfully". I have rebooted afterwards. Grub2 only shows 2 fedora 39 options

I have looked at my /etc/grub.d/grub.cfg file. It does look like 41 is trying to source its config from a separate file (/boot/grub2/custom.cfg), while 39 puts its config menuitem options directly on the file:

Output of `sudo cat /boot/grub2/grub.cfg`

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/08_fallback_counting ###
insmod increment
# Check if boot_counter exists and boot_success=0 to activate this behaviour.
if [ -n "${boot_counter}" -a "${boot_success}" = "0" ]; then
  # if countdown has ended, choose to boot rollback deployment,
  # i.e. default=1 on OSTree-based systems.
  if  [ "${boot_counter}" = "0" -o "${boot_counter}" = "-1" ]; then
    set default=1
    set boot_counter=-1
  # otherwise decrement boot_counter
  else
    decrement boot_counter
  fi
  save_env boot_counter
fi
### END /etc/grub.d/08_fallback_counting ###

### BEGIN /etc/grub.d/10_linux ###
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=boot 813B-5EFC

# This section was generated by a script. Do not modify the generated file - all changes
# will be lost the next time file is regenerated. Instead edit the BootLoaderSpec files.
#
# The blscfg command parses the BootLoaderSpec files stored in /boot/loader/entries and
# populates the boot menu. Please refer to the Boot Loader Specification documentation
# for the files format: https://systemd.io/BOOT_LOADER_SPECIFICATION/.

# The kernelopts variable should be defined in the grubenv file. But to ensure that menu
# entries populated from BootLoaderSpec files that use this variable work correctly even
# without a grubenv file, define a fallback kernelopts variable if this has not been set.
#
# The kernelopts variable in the grubenv file can be modified using the grubby tool or by
# executing the grub2-mkconfig tool. For the latter, the values of the GRUB_CMDLINE_LINUX
# and GRUB_CMDLINE_LINUX_DEFAULT options from /etc/default/grub file are used to set both
# the kernelopts variable in the grubenv file and the fallback kernelopts variable.
if [ -z "${kernelopts}" ]; then
  set kernelopts="root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 ro rootflags=subvol=root/ostree/deploy/default/deploy/9307d69d321f00528072b25fedcbc6c9862473df97a09f4a77a084ab7d410a4b.0 rd.luks.options=discard rhgb quiet "
fi

insmod blscfg
blscfg
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/10_reset_boot_success ###
# Hiding the menu is ok if last boot was ok or if this is a first boot attempt to boot the entry
if [ "${boot_success}" = "1" -o "${boot_indeterminate}" = "1" ]; then
  set menu_hide_ok=1
else
  set menu_hide_ok=0 
fi
# Reset boot_indeterminate after a successful boot
if [ "${boot_success}" = "1" ] ; then
  set boot_indeterminate=0
# Avoid boot_indeterminate causing the menu to be hidden more than once
elif [ "${boot_indeterminate}" = "1" ]; then
  set boot_indeterminate=2
fi
# Reset boot_success for current boot 
set boot_success=0
save_env boot_success boot_indeterminate
### END /etc/grub.d/10_reset_boot_success ###

### BEGIN /etc/grub.d/12_menu_auto_hide ###
if [ x$feature_timeout_style = xy ] ; then
  if [ "${menu_show_once}" ]; then
    unset menu_show_once
    save_env menu_show_once
    set timeout_style=menu
    set timeout=60
  elif [ "${menu_auto_hide}" -a "${menu_hide_ok}" = "1" ]; then
    set orig_timeout_style=${timeout_style}
    set orig_timeout=${timeout}
    if [ "${fastboot}" = "1" ]; then
      # timeout_style=menu + timeout=0 avoids the countdown code keypress check
      set timeout_style=menu
      set timeout=0
    else
      set timeout_style=hidden
      set timeout=1
    fi
  fi
fi
### END /etc/grub.d/12_menu_auto_hide ###

### BEGIN /etc/grub.d/14_menu_show_once ###
if [ x$feature_timeout_style = xy ]; then
  if [ "${menu_show_once_timeout}" ]; then
    set timeout_style=menu
    set timeout="${menu_show_once_timeout}"
    unset menu_show_once_timeout
    save_env menu_show_once_timeout
  fi
fi
### END /etc/grub.d/14_menu_show_once ###

### BEGIN /etc/grub.d/15_ostree ###
menuentry 'Fedora Linux 39.20240419.0 (Bazzite) (ostree:0)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-0-8324ea68-fada-463f-aaa0-a1dfa764c034' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
linux16 /ostree/default-57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/vmlinuz-6.7.12-202.fsync.fc39.x86_64 rd.luks.options=discard rhgb quiet root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 rootflags=subvol=root rw ostree=/ostree/boot.1/default/57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/0 gpu_sched.sched_policy=0
initrd16 /ostree/default-57f279771ffe042d1cc721e48fc86123e341e99345329eb0507eb3223ad0c94d/initramfs-6.7.12-202.fsync.fc39.x86_64.img
}
menuentry 'Fedora Linux 39.20240406.0 (Bazzite) (ostree:1)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-1-8324ea68-fada-463f-aaa0-a1dfa764c034' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 8324ea68-fada-463f-aaa0-a1dfa764c034
linux16 /ostree/default-27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/vmlinuz-6.7.11-202.fsync.fc39.x86_64 rd.luks.options=discard rhgb quiet root=UUID=e8f0fac1-baf3-46f1-95bd-f848c3868da5 rootflags=subvol=root rw ostree=/ostree/boot.1/default/27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/0 gpu_sched.sched_policy=0
initrd16 /ostree/default-27fb5e27df7c4b0c0aa7ef3005c0a655dab79bdcb957011603058de0e1725c2e/initramfs-6.7.11-202.fsync.fc39.x86_64.img
}
### END /etc/grub.d/15_ostree ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
if [ "$grub_platform" = "efi" ]; then
        menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
                fwsetup
        }
fi
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/35_fwupd ###
### END /etc/grub.d/35_fwupd ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg
fi
### END /etc/grub.d/41_custom ###

That seems to be part of the issue; assuming config_directory for grub means /boot/grub2, there's a /boot/grub2/grubenv file there, but there's no custom.cfg file:

sudo ls /boot/grub2
fonts  grub.cfg  grubenv

It definetly looks like there's a problem in writing to grub2, I got a warning when running rpm-ostree status (pasting that below, on the next section). It suggests running journalctl -b -1 -u ostree-finalize-staged.service, which I did:

Jan 25 23:18:15 fedora systemd[1]: Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jan 26 01:18:50 fedora systemd[1]: Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Jan 26 01:18:50 fedora ostree[29050]: Finalizing staged deployment
Jan 26 01:18:51 fedora ostree[29050]: Copying /etc changes: 24 modified, 1 removed, 93 added
Jan 26 01:18:51 fedora ostree[29050]: Copying /etc changes: 24 modified, 1 removed, 93 added
Jan 26 01:18:51 fedora ostree[29050]: Refreshing SELinux policy
Jan 26 01:18:55 fedora ostree[29050]: Refreshed SELinux policy in 3581 ms
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Finalized deployment
Jan 26 01:18:55 fedora ostree[29050]: bootfs is sufficient for calculated new size: 165.8 MB
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Jan 26 01:18:55 fedora ostree[29050]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.
Jan 26 01:18:55 fedora systemd[1]: Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Jan 26 01:18:55 fedora systemd[1]: ostree-finalize-staged.service: Consumed 4.521s CPU time.

There is a couple warnings there about a version mismatch (10.42 vs 10.44) which I don't know what to do with.

Output of rpm-ostree status

ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:3120d11f0690f815432fe21f5d61f1b633e4e1a0a0fb22541b6ebde3f40a1246
                  Version: 41.20250124 (2025-01-24T08:14:07Z)
                     Diff: 2128 upgraded, 5 downgraded, 403 removed, 668 added
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"' 

  ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:fe357496bb5ac67edd223e8cf27ed8d29384eb6948a082b98c265ec8a9d9ac90
                  Version: 39.20240419.0 (2024-04-19T17:04:05Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"' 

● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite:latest
                   Digest: sha256:7c9d16a34d8192df6ba824198c2e2343faa8108d9b53e949cebeed4d849e2395
                  Version: 39.20240406.0 (2024-04-07T01:29:50Z)
                Initramfs: '"-I /etc/crypttab /etc/modprobe.d/amdgpu.conf"'

Hardware

This is a desktop PC.

• CPU: AMD Ryzen 5 7600X 6-Core
• Motherboard: Asus TUF GAMING B650-PLUS
• Graphics card: Radeon RX 6650 XT Core 8GB GDDR6
• Hard Drive: WD_BLACK SN770 1TB

Extra information or context

No response

[jbtrystram]

Hello !
Can you show me the output of the following :
findmnt / and rpm-ostree version ?

And also :
journalctl -u ostree-finalize-staged.service --no-pager -o cat -b -1

[kikito]

Hi @jbtrystram , right away:

Output of findmt /:

$ findmt /

TARGET
  SOURCE                                                                                                                FSTYPE OPTIONS
/ /dev/nvme0n1p3[/root/ostree/deploy/default/deploy/9307d69d321f00528072b25fedcbc6c9862473df97a09f4a77a084ab7d410a4b.0] btrfs  rw,noatime,seclabel,compress-force=zstd:1,ssd,discard=async,space_cache=v2,commit=120,subvolid=257,subvol=/root

rpm-ostree version said unknown command, I suppose you meant rpm-ostree --version. Here it is:

$ rpm-ostree --version

rpm-ostree:
 Version: '2024.4'
 Git: 900d805f2116135469f31931250a4d14e2be44bb
 Features:
  - rust
  - compose
  - container
  - fedora-integration

And finally, here's journalctl:

$ journalctl -u ostree-finalize-staged.service --no-pager -o cat -b -1

Finished ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
Stopping ostree-finalize-staged.service - OSTree Finalize Staged Deployment...
Finalizing staged deployment
Copying /etc changes: 24 modified, 1 removed, 93 added
Copying /etc changes: 24 modified, 1 removed, 93 added
Refreshing SELinux policy
Refreshed SELinux policy in 3587 ms
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Finalized deployment
bootfs is sufficient for calculated new size: 165.8 MB
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
Regex version mismatch, expected: 10.42 2022-12-11 actual: 10.44 2024-06-07
error: Bootloader write config: grub2-mkconfig: Child process exited with code 127
ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
ostree-finalize-staged.service: Failed with result 'exit-code'.
Stopped ostree-finalize-staged.service - OSTree Finalize Staged Deployment.
ostree-finalize-staged.service: Consumed 4.536s CPU time.

I hope it helps, let me know if there's anything else I can provide. Thanks for your help!

[HikariKnight]

since youre updating from fedora 39 to fedora 41, did you read https://universal-blue.discourse.group/t/important-announcement-regarding-system-updates-action-needed/2689 ?

rebase to a bazzite version based on fedora 40 before july 2024 then run the script from the announcement, then rebase to latest and it should work possibly.
not sure about the mismatches you are getting though