S3 External Pillar Missing Data

[Linuturk]

Description of Issue

Pillar data stored in S3 isn't found for all minions, even after clearing cache and forcing a refresh. https://docs.saltstack.com/en/latest/ref/pillar/all/salt.pillar.s3.html

Setup

Configure salt master to use S3 as a pillar source:

ext_pillar:
  - git:
    - __env__ ssh://[email protected]/pillar.git:
      - pubkey: '/etc/salt/ssh/saltmaster-id_rsa.pub'
      - privkey: '/etc/salt/ssh/saltmaster-id_rsa'
  - s3:
      bucket: mys3bucket
      prefix: pillar

Write your pillar data to S3. We have a unique pillar setup for each minion in the format of minionid/init.sls. We currently have 2249 pillar folder/file combinations in the bucket.

top.sls:

{{saltenv}}:
  {{opts.id}}:
    - {{opts.id}}
    - ignore_missing: True

Steps to Reproduce Issue

Call pillar.items on a given minion and see that all the pillar data from S3 isn't present. Only the data from our other pillars.

Use saltutil.refresh_pillar return clean, and still show empty for the S3 data.

Use salt-run cache.pillar and see the S3 data isn't cached. Clear the cache with salt-run cache.clear_pillar and refresh again, and still don't see the data.

Notice that this data is available for some minions, but not all.

Versions Report

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 2.7.15+ (default, Oct  7 2019, 17:39:04)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1054-aws
         system: Linux
        version: Ubuntu 18.04 bionic

[Linuturk]

Looking at /var/cache we only have 999 items (including the top.sls) out of the more than 2000 expected. Seems this might be a pagination issue?

[Linuturk]

Snippet from the debug log:

[DEBUG   ] LazyLoaded s3.ext_pillar
[DEBUG   ] S3 bucket cache file /var/cache/salt/master/pillar_s3fs/mybucket-pillar-files.cache is not expired, mtime_diff=20.8665513992s, expiration=30s
[DEBUG   ] Reading buckets cache file
[DEBUG   ] S3 bucket retrieved pillars {...} << contains a list of 999 items, there are a lot more in the bucket.

[garethgreenaway]

@Linuturk Thanks for the report. Will check this out, seems like your thought about pagination might be correct.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[sagetherage]

Looks like this is merged @garethgreenaway can we removed the Pending Discussion label and close it?

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[sagetherage]

closing as fixed