[BUG] salt-ssh doesnt work with git_pillar

[alexxy]

Description

salt-ssh seems like doesnt work with git plillars (it can read top.sls but cannot read includes sls files)

Setup

salt running on physical machine using gitfs via pygit for states, formulas and pillars

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)

Steps to Reproduce the behavior

output running

salt-ssh -l all --hard-crash --refresh  'glpi*' pillar.items

[DEBUG   ] RETCODE glpi-2.ctrl: 0
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] pygit2 git_pillar_provider enabled
[TRACE   ] Loaded git_pillar as virtual git
[DEBUG   ] LazyLoaded git.ext_pillar
[DEBUG   ] Current fetch URL for git_pillar remote 'main git://bridge.ctrl/salt/pillar': git://bridge.ctrl/salt/pillar (desired: git://bridge.ctrl/salt/pillar)
[DEBUG   ] Current refspecs for git_pillar remote 'main git://bridge.ctrl/salt/pillar': ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'] (desired: ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*'])
[DEBUG   ] Current http.sslVerify for git_pillar remote 'main git://bridge.ctrl/salt/pillar': true (desired: true)
[DEBUG   ] Set update lock for git_pillar remote 'main git://bridge.ctrl/salt/pillar'
[DEBUG   ] Fetching git_pillar remote 'main git://bridge.ctrl/salt/pillar'
[DEBUG   ] git_pillar remote 'main git://bridge.ctrl/salt/pillar' is up-to-date
[DEBUG   ] Removed update lock for git_pillar remote 'main git://bridge.ctrl/salt/pillar'
[DEBUG   ] git_pillar is processing pillar SLS from /var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414 for pillar env 'main'
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] compile template: /var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414/top.sls
[DEBUG   ] Jinja search path: ['/var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414']
[DEBUG   ] Using importlib_metadata to load entry points
[PROFILE ] Time (in seconds) to render '/var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414/top.sls' using 'jinja' renderer: 0.025183677673339844
[DEBUG   ] Rendered data from file: /var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414/top.sls:
base:
  '*':
    - portage
  'outss*':
    - pum-outss


[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['portage']), ('outss*', ['pum-outss'])]))])
[PROFILE ] Time (in seconds) to render '/var/tmp/.root_d06d47_salt/running_data/var/cache/salt/minion/git_pillar/2da99e74f842c36e29c8f34a9766adc07aaae4254d689e8b82176776efe4d414/top.sls' using 'yaml' renderer: 0.0005822181701660156
[DEBUG   ] LazyLoaded confirm_top.confirm_top
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] compound_match: glpi-2.ctrl ? *
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match glpi-2.ctrl ? "*" => "True"
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded compound_match.match
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] compound_match: glpi-2.ctrl ? outss*
[DEBUG   ] LazyLoaded glob_match.match
[DEBUG   ] compound_match glpi-2.ctrl ? "outss*" => "False"
[DEBUG   ] Specified SLS 'portage' in environment 'base' was not found. This is likely caused by a git_pillar top file containing an environment other than the one for the branch in which it resides. Each git_pillar branch/tag must have its own top file.
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded pillar.items
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded nested.output
[TRACE   ] data = {'glpi-2.ctrl': {}}
glpi-2.ctrl:
    ----------
[TRACE   ] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Sending event: tag = salt/job/20220327173909975905/ret/glpi-2.ctrl; data = {'return': {}, 'id': 'glpi-2.ctrl', 'fun': 'pillar.items', 'jid': '20220327173909975905', '_stamp': '2022-03-27T17:39:13.007009'}
[DEBUG   ] Closing IPCMessageClient instance

Expected behavior
salt-ssh should return pillar like this (same setup but not using salt-ssh)

# salt 'glpi-2*' pillar.items
glpi-2.ctrl:
    ----------
    portage:
        ----------
        binrepos:
            http://binpkgs.ctrl
        compileflags:
            -O2 -pipe -march=cascadelake -mtune=cascadelake
        linguas:
            ru en
        mirrors:
            http://distfiles.ctrl
        python_targets:
            python3_9
        rsync:
            rsync://bridge.ctrl/gentoo-portage
        ruby_targets:
            ruby30

Versions Report

# salt --versions-report
Salt Version:
          Salt: 3003.3
 
Dependency Versions:
          cffi: 1.15.0
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: 1.4.1
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.3
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: 3.14.1
  pycryptodome: 3.14.1
        pygit2: 1.9.0
        Python: 3.9.9 (main, Mar  5 2022, 13:05:41)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 22.1.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: gentoo 2.7 
        locale: utf-8
       machine: x86_64
       release: 5.15.25-gentoo-dist
        system: Linux
       version: Gentoo 2.7 

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

What are your pillar sources and configuration?
Why are you expecting something different to what it gives you?

Have you checked this hint?

This is likely caused by a git_pillar top file containing an environment other than the one for the branch in which it resides. Each git_pillar branch/tag must have its own top file.

[alexxy]

Ok. Additional info:

Expected result

# salt 'glpi-2*' pillar.items
glpi-2.ctrl:
    ----------
    portage:
        ----------
        binrepos:
            http://binpkgs.ctrl
        compileflags:
            -O2 -pipe -march=cascadelake -mtune=cascadelake
        linguas:
            ru en
        mirrors:
            http://distfiles.ctrl
        python_targets:
            python3_9
        rsync:
            rsync://bridge.ctrl/gentoo-portage
        ruby_targets:
            ruby30

Git pillar config:

git_pillar_provider: pygit2
git_pillar_base: main

ext_pillar:
  - git:
    - main git://bridge.ctrl/salt/pillar

Pillar structure:

alexxy@bridge ~/salt/pillar $ git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
alexxy@bridge ~/salt/pillar $ ls -l
total 16
-rw-r--r-- 1 alexxy alexxy   9 Mar 24 08:06 README.md
-rw-r--r-- 1 alexxy alexxy 251 Mar 24 08:07 portage.sls
-rw-r--r-- 1 alexxy alexxy 722 Mar 24 08:07 pum-outss.sls
-rw-r--r-- 1 alexxy alexxy  56 Mar 25 11:35 top.sls

Contents of top.sls

base:
  '*':
    - portage
  'outss*':
    - pum-outss

Contents of portage.sls

portage:
  mirrors: http://distfiles.ctrl
  binrepos: http://binpkgs.ctrl
  rsync: rsync://bridge.ctrl/gentoo-portage
  compileflags: -O2 -pipe -march=cascadelake -mtune=cascadelake
  linguas: ru en
  ruby_targets: ruby30
  python_targets: python3_9

And other note: this pillar renders fine with salt-minion, but not with salt-ssh

[alexxy]

I also checked 3003.4 and 3004.1 both have same issue

[alexxy]

Looks like salt-ssh simply dont pack other pillars when sending data to minions

[max-arnold]

Also got bitten by this. The strange thing is that git_pillar was confirmed to work with salt-ssh earlier by @Ch3LL: #40007 (comment)

So I copied (not cloned!) the example repo contents into my own repo's main branch https://github.com/max-arnold/Ch3LLScripts

And it doesn't work with my config provided below. But when I replace the main branch (see the the <-- marks) with the master, it starts to work!

So it looks like the old-style master branch name is hardcoded somewhere (at least when it is used over salt-ssh).

# GitFS:
gitfs_provider: gitpython
gitfs_update_interval: 120
gitfs_global_lock: false
# file_roots: {}
file_roots:
  base:
    - /srv/salt

fileserver_backend:
  - roots
  - gitfs

gitfs_remotes:
  - https://github.com/max-arnold/Ch3LLScripts:
    - base: main                                       # <--
    - root: salt/states


# Git Pillar
# pillar_roots: {}
pillar_roots:
  base:
    - /srv/pillar

git_pillar_provider: gitpython
git_pillar_base: main                                  # <--
git_pillar_branch: main                                # <--
pillarenv_from_saltenv: true
ext_pillar_first: true
git_pillar_update_interval: 120
git_pillar_global_lock: false

ext_pillar:
  - git:
    - main https://github.com/max-arnold/Ch3LLScripts: # <--
      - root: salt/pillars

[vemilyus]

It's been quite some time, and this bug is marked as highly severe. Is there a timeline for this fix?

This issue still persists in version 3005.1.

[lkubb]

I suspect that this is at least in part caused by the pillar compilation running with incorrect __opts__, especially cachedir and __role. salt-ssh merges minion and master opts for pillar compilation, with the minion opts having priority currently. The resulting cache dir is thus not the usual one and the git repos will be re-initialized. Since __role is set to minion, it will additionally do some extra stuff for masterless minions.

Thus this likely has the same root cause as #60002 and should be fixed by #65484, but I'm not 100% sure. Maybe someone experiencing this issue can check with the mentioned fix.

[max-arnold]

@lkubb I can confirm that it works in 3006.5. Thanks a lot, this is an awesome fix!

@alexxy Could you please verify that on your side and close the issue if it works for you as well?

[dwoz]

Closing as fixed per #65067 (comment)