4.2: user_salt isn't working

[rapenne-s]

How to file a helpful issue

Qubes OS release

Fresh install of 4.2-RC3

Brief summary

I installed 4.2-RC3, enabled user-dirs in salt, imported my salt repo which work on other Qubes OS deployments (that are not fresh 4.2-RC3 but upgraded from 4.1), it fails in weird ways.

dom0 related configuration are working 50% of the time

non-dom0 configuration aren't working, qubesctl report that no top file are matching their name, while it works on other computers. Although, adding saltenv=user make it work!

Steps to reproduce

• Enable user-dirs with qubesctl state.sls qubes.user-dirs

• Modify /srv/user_salt/top.sls to look like

user:
  'dom0':
    - qubes-vm

• Create /srv/user_salt/qubes-vm.sls with this content:

create-qubes-www:
  qvm.vm:
    - name: WWW
    - present:
      - template: fedora-38
      - label: orange
    - prefs:
      - label: orange

• Run qubesctl state.highstate

First run fails

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[ERROR   ] State 'qvm.vm' was not found in SLS 'qubes-vm'
Reason: 'qvm.vm' is not available.

local:
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: False
     Comment: State 'qvm.vm' was not found in SLS 'qubes-vm'
              Reason: 'qvm.vm' is not available.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing

2nd run works

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
----------
          ID: topd-always-passes
    Function: test.succeed_without_changes
        Name: foo
      Result: True
     Comment: Success!
     Started: 10:23:00.000115
    Duration: 0.588 ms
     Changes:   
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'WWW' already exists.
              
              ====== ['prefs'] ======
              [SKIP] label              : orange
     Started: 10:23:00.031787
    Duration: 413.524 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 414.112 ms

3rd run doesn't work

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
[ERROR   ] State 'qvm.vm' was not found in SLS 'qubes-vm'
Reason: 'qvm.vm' is not available.

local:
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: False
     Comment: State 'qvm.vm' was not found in SLS 'qubes-vm'
              Reason: 'qvm.vm' is not available.
     Changes:   

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing

4th run works

[root@dom0 srv]# qubesctl --targets dom0 --show-output state.apply
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
----------
          ID: topd-always-passes
    Function: test.succeed_without_changes
        Name: foo
      Result: True
     Comment: Success!
     Started: 10:23:15.758674
    Duration: 0.636 ms
     Changes:   
----------
          ID: create-qubes-www
    Function: qvm.vm
        Name: WWW
      Result: True
     Comment: ====== ['present'] ======
              [SKIP] A VM with the name 'WWW' already exists.
              
              ====== ['prefs'] ======
              [SKIP] label              : orange
     Started: 10:23:15.788875
    Duration: 328.364 ms
     Changes:   

Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 329.000 ms

Reproducer for template

user_salt for templates is completely broken:

• Enable user-dirs with qubesctl state.sls qubes.user-dirs

• Modify /srv/user_salt/top.sls to look like

user:
  'fedora-38':
    - package

• Create /srv/user_salt/package.sls with this content:

jq:
  pkg.installed

• Run qubesctl --targets fedora-38 --skip-dom0 --show-output state.highstate

[root@dom0 srv]# qubesctl --targets fedora-38 --skip-dom0  --show-output state.highstate
fedora-38:
      An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
  [WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
  [ERROR   ] An Exception occurred while executing state.highstate: Unable to render top file. No targets found.
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.

It's possible to fix the command by adding saltenv=user:

• Run qubesctl --targets fedora-38 --skip-dom0 --show-output state.highstate saltenv=user

[root@dom0 srv]# qubesctl --targets fedora-38 --skip-dom0  --show-output state.highstate saltenv=user
fedora-38:
  ----------
            ID: jq
      Function: pkg.installed
        Result: True
       Comment: All specified packages are already installed
       Started: 10:36:50.037392
      Duration: 3369.414 ms
       Changes:   
  
  Summary for fedora-38
  ------------
  Succeeded: 1
  Failed:    0
  ------------
  Total states run:     1
  Total run time:   3.369 s
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.

Expected behavior

It works

Actual behavior

mix of half working for dom0 and not working for non-dom0

[rapenne-s]

Hold on, it may be a mistake of mine, I'm trying to figure what's going on here 🙏

edit: it often fails with the error above, but sometimes it just works and create the qube 🤷‍♀️

[rapenne-s]

Ok! I've been able to make a reproducer for dom0 and fedora 38 template.

I compared my /srv and /etc/salt files with a working Qubes OS running 4.2-RC3, they are 100% identical, yet it works on the other computers.

The line [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable seems related to the issue.

[DemiMarie]

@rapenne-s does sudo qubesctl saltutil.sync_all help? It did for me. I’m not sure if that should be necessary, though.

[rapenne-s]

@rapenne-s does sudo qubesctl saltutil.sync_all help? It did for me. I’m not sure if that should be necessary, though.

this didn't change anything

[DemiMarie]

Does --target=fedora-38 (instead of --targets) help?

[rapenne-s]

Does --target=fedora-38 (instead of --targets) help?

no, while qubesctl man pages states --targets foo,bar,stuff is the correct syntax, for some reason --target=foo or --target foo also work, but they all produce the same result in that case

[rapenne-s]

@andrewdavidwong isn't this a blocker?

[andrewdavidwong]

Sorry, I'm not very familiar with Salt. Can you explain in simple terms why it's a blocker? For example, which use cases does it break that will affect most users?

[gonzalo-bulnes]

@andrewdavidwong When using Salt to automate configuration, you've got the choice of placing your personal "files" in one of two environments: base or user. Both exist in the default setup of Salt in Qubes OS.

• base is where the Salt "files" created by the Qubes OS team live, you may put your own there, but there is a risk that a system update will overwrite them
• the user environment is guaranteed not to be overwritten by system updates

If you're like me, you do what @rapenne-s did: enable the so-called "user directories" and write your stuff in the user environment.

Is that most users? (I don't know.) Is that most users who use Salt? (I don't know either.) What I know is that a few of us would see our workflows reliability go down significantly* by not being able to prevent conflicts between our personal automation efforts and the base ones. Does this make sense?

* At some point in the history of Qubes OS, this was determined to be significant enough for the user environment to be set up in the first place, and qubes.user-dirs to be written for the purpose of making enabling those user directories a very straightforward operation.

[andrewdavidwong]

@gonzalo-bulnes: Thanks for the explanation! I suppose if the user environment is supposed to be "guaranteed" to work in the manner you describe (and currently doesn't), then that would be a pretty strong reason for considering this a blocker, but it's @marmarek's call either way.

[rapenne-s]

thanks @gonzalo-bulnes for the explanation! I'd add that a lot of community work is provided through Salt (like sys-vpn), which is broken in this case.

[rapenne-s]

if this helps, when looking at pillar items on both my systems, I found a huge difference:

4.1 -> 4.2-RC3 system that work

# qubesctl --target TheQube --skip-dom0 --show-output pillar.items
TheQube:
      ----------
      qubes:
          ----------
          netvm:
              sys-firewall
          template:
              fedora-38
          type:
              app
      salt-modules:
          ----------
          topd:
              ----------
              topd_base_pillar:
                  /srv/pillar
              topd_base_state:
                  /srv/salt
              topd_dir_name:
                  _tops

The fresh 4.2-RC3 which have issues

# qubesctl --target TheQube --skip-dom0 --show-output pillar.items
TheQube:
      ----------
      qubes:
          ----------
          netvm:
              sys-firewall
          template:
              fedora-38
          type:
              app

[rapenne-s]

Investigating further, it seems qubesctl saltutil.sync_all isn't idempotent on the fresh 4.2-RC3, while the upgraded one doesn't show this random behavior.

When running the qubesctl saltutil.sync_all and then qubesctl --target TheQube --skip-dom0 --show-output pillar.items

• sometimes it returns only the qubes structure
• sometimes only the salt-modules structure
• sometimes both

[rapenne-s]

I found a fix for the issue, but is it really that?

When comparing my two salt directories, the system with the issue was missing a link in /src/_tops/base/qubes.user-dirs.top to /srv/salt/qubes/user-dirs.top, despite running qubesctl state.sls qubes.user-dirs (this command is in in /srv/salt/qubes/user-dirs.sls)

Making the link allows me to deploy using my user defined salt states, removing the link make the error appearing again.

[DemiMarie]

That could well be it @rapenne-s!

[feskehau]

Had the same issue on 4.2-rc3

ln -s /srv/salt/qubes/user-dirs.top /srv/salt/_tops/base/user-dirs.top

Solved it for me. Thanks:)

edit, corrected path...

[marmarek]

I just checked on a fresh 4.1 installation, and it is this way there too, so it isn't regression.

The simplest way to create those dirs is to run qubesctl state.apply qubes.user-dirs. That probably needs to be documented somewhere (on https://www.qubes-os.org/doc/salt/ ?). But it definitely doesn't qualify as release blocker.

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package mgmt-salt) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

[rapenne-s]

I leave this in case it's useful to someone one day, I had the error [CRITICAL] Specified ext_pillar interface qvm_prefs is unavailable on a working 4.2 (I certainly did something wrong), preventing dom0 to be updated from the GUI updater.

The recommend command qubesctl saltutil.sync_all solved it.

[zaz]

This issue remains for me on a fresh install of Qubes 4.2.3 in spite of qubes-mgmt-salt PR42 having been merged. Applying qubes-mgmt-salt-base-config PR11 fixes it for me.

[marmarek]

@zaz have you done qubesctl state.apply qubes.user-dirs? Did it failed without the change you mentioned? If so, what was the result?

[zaz]

@marmarek Yes. I ran that command and my SALT states in /srv/user_salt/ would not run until the link is created:

sudo qubesctl --all state.highstate
[WARNING ] top_file_merging_strategy is set to 'merge' and multiple top files were found. Merging order is not deterministic, it may be desirable to either set top_file_merging_strategy to 'same' or use the 'env_order' configuration parameter to specify the merging order.
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing
[user@dom0 ~]$ sudo qubesctl --all state.highstate saltenv=user
local:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or master_tops data matches found. Please see master log for details.
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   0.000 ms
DOM0 configuration failed, not continuing

With the modified user-dirs.sls, sudo qubesctl --all state.highstate works without saltenv=user, but with saltenv=user yields the same error ("No Top file or master_tops data matches found.")

[marmarek]

@zaz do you have anything in /srv/user_salt/top.sls ? Or maybe you have some other top files?

[zaz]

@marmarek Yes

user:
  fedora-4?-xfce:
    - common
    - fedora-pkgs
  debian:
    - common
    - debian-pkgs
  sys-net:
    - restart-network

And these SALT states are applied when the link exists.