Template updates fail with a VMRootShell permission error

[ghost]

Qubes OS release

Qubes 4.1 RC2

Brief summary

I have upgraded QubesOS from 4.0 to 4.1 RC2 without any issues.

But now if I run the Qubes Updater dom0 can be updated, but if the templates are updated I run into a permission issue.

The management vm disp-mgmt-debian-11 is reported as started, the template vm never seems to get startet and shortly after I get the error message box:

Denied: qubes.VMRootShell
Denied qubes.VMRootShell from disp-mgmt-debian-11 to debian-11

The output of the updater looks as follow:

Updating debian-11

Error on updating debian-11: Command '['sudo', 'qubesctl', '--skip-dom0', '--targets=debian-11', '--show-output', 'state.sls', 'update.qubes-vm']' returned non-zero exit status 20.
debian-11:
      ----------
      _error:
          Failed to return clean data
      retcode:
          126
      stderr:
          Request refused
      stdout:

The same happens if I run the update command outside of Qubes Update.

Steps to reproduce

1. Upgrade from QubesOS 4.0 to 4.1
2. Run Qubes Update against one template
3. Check the results

Expected behavior

Qubes Update works as expected without permission errors

Actual behavior

Qubes Update can not update the templates because of permission errors

[marmarek]

Do you get a notification about failed template start (not enough memory maybe)? If not, can you look into journalctl for the line about the denial - you should find more details about it; just search for qubes.VMRootShell

[ghost]

Many thanks for your help and sorry for not checking journalctl too.

journalctl showed me that the permission was denied because of /etc/qubes-rpc/policy/qubes.VMRootShell.

This file did had quite some entries for disp-mgmt-* vms and a deny for anyvm at the bottom.

Since the file was not tracked by an rpm package I moved it away and now it works.

From my side the bug can be closed but maybe this is an upgrading issue. I do not remember to have created or edited this file.

[qtpies]

I have this exact same issue in 4.1.0. Solution works.

[tlaurion]

I confirm this too after upgrading from 4.0.4 with qubes-dist-upgrade tool just now.

Content of qubes.VMRootShell:
disp-mgmt-fedora-33 fedora-33 allow,user=root
Then $anyvm $anyvm deny

So of course, it fails.

Even though previous 4.0.4 had fedora-34. So 4.0.4 was not using it? Or the upgrade script created it? Seems like there is some bug. I cloned 4.0.4 OEM disk image prior of testing this so if you need more info I can feed, please ask.

Is that policy still under use? How is it maintained across template upgrades? Backed up the file and removed it to proceed with template upgrades.

@marmarek ? @fepitre?

[mfc]

encountered same issue with in-place R4.0->R4.1 upgrade.

backing up /etc/qubes-rpc/policy/qubes.VMRootShell in dom0 and deleting it solved issue as described by @ekaflaer

[tlaurion]

Hmmm I will try to make more time into troubleshooting this and #7114 which are blockers to actually push users into upgrading before Whonix EOL of 03/22 on Qubes 4.0.4

@adrelanos @marmarek: Delays are really short for users. Really surprised by 4.1 release having came so fast. Now we expect users to reinstall and restore backup on top of 4.1 it seems.

[tlaurion]

qubes-dist-upgrade-4.0.4-1.fc25.noarch
Q4.0 fully updated rpm -qa
qubes-dist-upgrade logs

[tlaurion]

Confirm that removing /etc/qubes-rpc/policy/qubes.VMRootShell fixes this issue.

[tlaurion]

@marmarek : What should users to with that file /etc/qubes-rpc/policy/qubes.VMRootShell file containing:

disp-mgmt-fedora-33 fedora-33 allow,user=root
Then $anyvm $anyvm deny

Seems like it is an artifact not updated by anything anymore?

[marmarek]

While the file indeed shouldn't be left there, it shouldn't break updates. The fact it does is a bug.

[tlaurion]

@marmarek depending on timeline of that being uploaded in repos, I recommend that file being removed by qubes-dist-upgrade script since a lot of users will experience that issue?

[qubesos-bot]

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.1 testing repository for the CentOS centos-stream8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package qubes-mgmt-salt-4.1.14-1.fc32) has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.1 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package qubes-mgmt-salt-4.1.14-1.fc32) has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-mgmt-salt_4.1.14-1 has been pushed to the r4.1 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing buster-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-mgmt-salt_4.1.16-1+deb10u1 has been pushed to the r4.1 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update