Qubes Updater silent failure while updating debian-10 TemplateVM #5705

emkll · 2020-03-04T19:33:31Z

Qubes OS version
Qubes release 4.0 (R4.0)

Affected component(s) or functionality
Qubes Updater / update/qubes-vm.sls

Brief summary
Qubes Updater fails to update Debian-based Template

To Reproduce

Open Qubes updater
Select debian-10
Click next
or
run sudo qubesctl --show-output --skip-dom0 --targets debian-10 state.sls update.qubes-vm in dom0.

Expected behavior

Packages inside the VM should be upgraded, and return OK
Otherwise, failed upgrade operations should return a failure message success in the UI

Actual behavior

Packages in the VM are not upgraded, and output of the qubesctl command does not contain OK
UI suggests the upgrade was successfully completed (qubesctl command returns 0), even though a Salt error occurred in the Template

What appears to be relevant output in debian-10 template:

Mar  4 14:18:21 localhost systemd[1]: Started Session c14 of user root.
Mar  4 14:18:22 localhost qubes.VMRootShell-disp-mgmt-debian-10: SALT_ARGV: ['/usr/bin/python3', '/var/tmp/.root_62a99a_salt/salt-call', '--retcode-passthrough', '--local', '--metadata', '--out', 'json', '-l', 'quiet', '-c', '/var/tmp/.root_62a99a_salt', '--', 'test.opts_pkg']
Mar  4 14:18:22 localhost qubes.VMRootShell-disp-mgmt-debian-10: _edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878
Mar  4 14:18:25 localhost systemd[1]: session-c14.scope: Succeeded.
Mar  4 14:18:25 localhost qrexec-agent[481]: send exit code 0
Mar  4 14:18:25 localhost qrexec-agent[481]: pid 1694 exited with 0
Mar  4 14:18:25 localhost qrexec-agent[481]: eintr
Mar  4 14:18:26 localhost qrexec-agent[481]: executed root:QUBESRPC qubes.VMRootShell disp-mgmt-debian-10 pid 1793
Mar  4 14:18:26 localhost systemd[1]: Started Session c15 of user root.
Mar  4 14:18:26 localhost systemd[1]: session-c15.scope: Succeeded.
Mar  4 14:18:26 localhost qrexec-agent[481]: send exit code 0
Mar  4 14:18:26 localhost qrexec-agent[481]: pid 1793 exited with 0
Mar  4 14:18:26 localhost qrexec-agent[481]: eintr
Mar  4 14:18:26 localhost qrexec-agent[481]: executed root:QUBESRPC qubes.VMRootShell disp-mgmt-debian-10 pid 1807
Mar  4 14:18:26 localhost systemd[1]: Started Session c16 of user root.
Mar  4 14:18:26 localhost qubes.VMRootShell-disp-mgmt-debian-10: SALT_ARGV: ['/usr/bin/python3', '/var/tmp/.root_62a99a_salt/salt-call', '--retcode-passthrough', '--local', '--metadata', '--out', 'json', '-l', 'quiet', '-c', '/var/tmp/.root_62a99a_salt', '--', 'state.pkg', '/var/tmp/.root_62a99a_salt/salt_state.tgz', 'test=None', 'pkg_sum=89f155d29e841fb86d260190219775a6bfafa1aeab4087303dfd9e763ca06cf5', 'hash_type=sha256']
Mar  4 14:18:26 localhost qubes.VMRootShell-disp-mgmt-debian-10: _edbc7885e4f9aac9b83b35999b68d015148caf467b78fa39c05f669c0ff89878
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10: Traceback (most recent call last):
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:   File "/var/tmp/.root_62a99a_salt/salt-call", line 27, in <module>
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:     salt_call()
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:   File "/var/tmp/.root_62a99a_salt/pyall/salt/scripts.py", line 445, in salt_call
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:     client.run()
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:   File "/var/tmp/.root_62a99a_salt/pyall/salt/cli/call.py", line 57, in run
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:     caller.run()
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:   File "/var/tmp/.root_62a99a_salt/pyall/salt/cli/caller.py", line 119, in run
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:     ret = self.call()
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:   File "/var/tmp/.root_62a99a_salt/pyall/salt/cli/caller.py", line 232, in call
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10:     func.__module__].__context__.get('retcode', 0)
Mar  4 14:18:31 localhost qubes.VMRootShell-disp-mgmt-debian-10: KeyError: 'salt.loaded.int.module.state'
Mar  4 14:18:32 localhost qrexec-agent[481]: send exit code 1
Mar  4 14:18:32 localhost qrexec-agent[481]: pid 1807 exited with 1
Mar  4 14:18:32 localhost systemd[1]: session-c16.scope: Succeeded.
Mar  4 14:18:32 localhost qrexec-agent[481]: eintr

Screenshots

Additional context

Installing qubes 4.0.3 from scratch, I could not immediately reproduce, it seems like an issue with existing templates.
For reference, the version of qubes-template-debian-10.noarch I am using is 4..1-201905201854

Solutions you've tried

Running apt upgrade directly on the VM itself does work, it seems like the issue is limited to salt

Relevant documentation you've consulted
none

Related, non-duplicate issues
none

The text was updated successfully, but these errors were encountered:

redshiftzero · 2020-03-04T21:56:30Z

possibly related: saltstack/salt#56131

emkll · 2020-03-04T22:02:21Z

I think the ticket linked by @redshiftzero is very likely related:

Based on my local testing, on a freshly installed Qubes 4.0.3 machine:

On a fully patched Fedora-30 template, the salt version is 3000-2. I have downgraded the salt package (via sudo dnf downgrade salt, which also downgrades salt-ssh) to version 2019.2.0-1.fc30. After downgrading this package, the updater (and Salt) work as expected.

conorsch · 2020-03-05T02:38:58Z

Worth noting that using F29 (warning: Fedora29 is EOL) for the mgmt vm also resolves, due to the implicit version downgrade. So setting qvm-prefs default-mgmt-dvm template fedora-29, assuming you still have qubes-template-fedora-29, lets qubesctl run successfully again. Sharing this info less as a workaround and more because I'm unsure what most folks have for the default mgmt dvm template. The last time I performed a clean install of the OS was Qubes 4.0.1, and if memory serves, I altered the value to F30 while removing F29 templates across the board a few months back.

marmarek · 2020-03-05T19:47:39Z

Looks like this wants backporting. Note version difference - apparently failing without python-pip package is a common pattern in salt...

DemiMarie · 2020-03-07T18:41:20Z

Should we install the python-pip package?

marmarek · 2020-03-07T19:01:41Z

This is exactly what the commit I pointed in #5705 (comment) does.

marmarek · 2020-03-08T14:41:16Z

If I read this correctly, we'd need python3-pip in every VM that is managed by salt, not only where salt-ssh is running. This means a simple dependency on a salt-related package isn't enough.

On the other hand, the failing code applies only to states using "onlyif" keyword - if I remove them, the update works. Since the salt bug will probably take some more time to fix (PR with the fix is there already, but then it would need to be merged, released and packaged into Fedora), I'd go with avoiding "onlyif" in the update formula. Fortunately, it is used only to apply updates that are incorporated into new templates already, so I can simply remove those entries.

salt-ssh may exit with code 0 even if applying state failed. One of such cases is saltstack/salt#56131. Add a heuristic that checks output data - if it's empty, report it as a failure for state.* command, as those always produce some output (changes summary). This converts silent failure into not-so-silent one (although there is still no details why it failed). Fixes QubesOS/qubes-issues#5705

eloquence · 2020-03-13T17:39:50Z

Thanks for the quick response & PRs, Marek. This is a critical issue for the SecureDrop Workstation pilot from our perspective as it prevents important system updates from succeeding, and we'd obviously prefer not to manually downgrade packages for security reasons. Is there any way we can help to get this over the finish line?

marmarek · 2020-03-13T17:56:39Z

Main blocker is testing capacity (openqa busy on other set of PRs already). If you could test those two PRs in your environment too, that would help a lot.

xet7 · 2020-03-14T23:23:17Z

When I update Qubes VMs, after updates it still shows that all did not update, with green arrow icon.

To get Debian 10 template updated, I need to do manually:

sudo apt update
sudo apt -y dist-upgrade
sudo apt clean
sudo apt autoclean
sudo apt -y autoremove

For Fedora 30 templates, I need to do manually:

sudo dnf upgrade
sudo dnf clean all

Would it be possible to change update scripts so that all this cleanup would also be run automatically?

This is broken in salt 3000 (saltstack/salt#56131) currently shipped in Fedora 30. Since those states apply changes already incorporated in all released templates (and latest stable installation image), simply drop them to unbreak updates. Fixes QubesOS/qubes-issues#5705 (cherry picked from commit e32dc3f)

qubesos-bot · 2020-03-16T03:40:08Z

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-update-4.0.9-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot · 2020-03-16T03:40:30Z

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-update-4.1.3-1.fc31 has been pushed to the r4.1 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot · 2020-03-25T00:41:26Z

Automated announcement from builder-github

The package qubes-mgmt-salt_4.0.21-1 has been pushed to the r4.0 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing stretch-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2020-03-28T03:17:38Z

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.1 testing repository for the CentOS centos7 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.1-current-testing

Changes included in this update

qubesos-bot · 2020-04-29T02:27:15Z

Automated announcement from builder-github

The package qubes-mgmt-salt-dom0-update-4.1.3-1.fc31 has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

qubesos-bot · 2020-05-07T02:26:57Z

Automated announcement from builder-github

The package qubes-mgmt-salt-4.0.22-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

qubesos-bot · 2020-05-07T02:37:48Z

Automated announcement from builder-github

The package qubes-mgmt-salt_4.0.22-1+deb9u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

qubesos-bot · 2020-05-07T03:12:31Z

Automated announcement from builder-github

The package mgmt-salt has been pushed to the r4.0 stable repository for the CentOS centos7 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

emkll added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug labels Mar 4, 2020

emkll mentioned this issue Mar 4, 2020

update.qubes-vm (and updater) salt not updating nightlies in dev env freedomofpress/securedrop-workstation#485

Closed

andrewdavidwong added C: mgmt P: major Priority: major. Between "default" and "critical" in severity. and removed P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Mar 5, 2020

andrewdavidwong added this to the Release 4.0 updates milestone Mar 5, 2020

marmarek self-assigned this Mar 5, 2020

marmarek mentioned this issue Mar 8, 2020

Drop states using 'onlyif' condition QubesOS/qubes-mgmt-salt-dom0-update#9

Merged

marmarek mentioned this issue Mar 8, 2020

Report empty output as a failure QubesOS/qubes-mgmt-salt#15

Merged

marmarek closed this as completed in marmarek/qubes-mgmt-salt-dom0-update@e32dc3f Mar 15, 2020

qubesos-bot added the r4.0-dom0-cur-test label Mar 16, 2020

qubesos-bot mentioned this issue Mar 16, 2020

mgmt-salt-dom0-update v4.0.9 (r4.0) QubesOS/updates-status#1697

Closed

qubesos-bot added the r4.1-dom0-cur-test label Mar 16, 2020

qubesos-bot mentioned this issue Mar 16, 2020

mgmt-salt-dom0-update v4.1.3 (r4.1) QubesOS/updates-status#1698

Closed

qubesos-bot added the r4.0-buster-cur-test label Mar 25, 2020

qubesos-bot added the r4.0-stretch-cur-test label Mar 25, 2020

qubesos-bot added the r4.1-centos7-cur-test label Mar 28, 2020

qubesos-bot mentioned this issue Apr 24, 2020

mgmt-salt v4.0.22 (r4.0) QubesOS/updates-status#1757

Closed

qubesos-bot added r4.1-dom0-stable and removed r4.1-dom0-cur-test labels Apr 29, 2020

qubesos-bot added r4.0-bullseye-stable r4.0-buster-stable and removed r4.0-bullseye-cur-test r4.0-buster-cur-test labels May 7, 2020

qubesos-bot added r4.0-stretch-stable r4.0-fc29-stable r4.0-fc30-stable r4.0-fc31-stable and removed r4.0-stretch-cur-test r4.0-fc29-cur-test r4.0-fc30-cur-test r4.0-fc31-cur-test labels May 7, 2020

qubesos-bot added r4.0-centos7-stable and removed r4.0-centos7-cur-test labels May 7, 2020

qubesos-bot added r4.1-fc29-stable r4.1-fc30-stable and removed r4.1-fc29-cur-test r4.1-fc30-cur-test labels Jun 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Qubes Updater silent failure while updating debian-10 TemplateVM #5705

Qubes Updater silent failure while updating debian-10 TemplateVM #5705

emkll commented Mar 4, 2020

redshiftzero commented Mar 4, 2020

emkll commented Mar 4, 2020

conorsch commented Mar 5, 2020

marmarek commented Mar 5, 2020

DemiMarie commented Mar 7, 2020

marmarek commented Mar 7, 2020

marmarek commented Mar 8, 2020

eloquence commented Mar 13, 2020

marmarek commented Mar 13, 2020

xet7 commented Mar 14, 2020

qubesos-bot commented Mar 16, 2020

qubesos-bot commented Mar 16, 2020

qubesos-bot commented Mar 25, 2020

qubesos-bot commented Mar 28, 2020

qubesos-bot commented Apr 29, 2020

qubesos-bot commented May 7, 2020

qubesos-bot commented May 7, 2020

qubesos-bot commented May 7, 2020

Qubes Updater silent failure while updating debian-10 TemplateVM #5705

Qubes Updater silent failure while updating debian-10 TemplateVM #5705

Comments

emkll commented Mar 4, 2020

redshiftzero commented Mar 4, 2020

emkll commented Mar 4, 2020

conorsch commented Mar 5, 2020

marmarek commented Mar 5, 2020

DemiMarie commented Mar 7, 2020

marmarek commented Mar 7, 2020

marmarek commented Mar 8, 2020

eloquence commented Mar 13, 2020

marmarek commented Mar 13, 2020

xet7 commented Mar 14, 2020

qubesos-bot commented Mar 16, 2020

qubesos-bot commented Mar 16, 2020

qubesos-bot commented Mar 25, 2020

qubesos-bot commented Mar 28, 2020

qubesos-bot commented Apr 29, 2020

qubesos-bot commented May 7, 2020

qubesos-bot commented May 7, 2020

qubesos-bot commented May 7, 2020