[ERROR ] Got a bad pillar from master, type str, expecting dict: #27792

tiger-seo · 2015-10-08T11:59:16Z

after upgrading to 2015.5.5 got this error:
[ERROR ] Got a bad pillar from master, type str, expecting dict:
on 2015.5.3 it works as expected

The text was updated successfully, but these errors were encountered:

jfindlay · 2015-10-09T20:03:09Z

@tiger-seo, thanks for the report. Is it possible for you to provide a minimal test case to demonstrate this problem?

sjwoodr · 2015-10-19T18:04:20Z

I see this too in the minion logs but I have no idea when it started or what caused it.... which I realize isn't very useful. This is with 2015.5.2 minions.

sjwoodr · 2015-10-19T18:33:13Z

@jfindlay --

Update: I figured it out.... I was not out of disk space but the master cache had exhausted 100% of the inodes for the /var filesystem. Solution:

stop the salt master
rm -rf /var/cache/salt/master
restart the salt master
possibly also restart the minion(s)

jfindlay · 2015-10-19T22:34:59Z

Excellent, thanks @sjwoodr. I have a suspicion this occult error message may be masking more than one problem. At least in the case you have found, I think we can handle and message a more useful error.

tiger-seo · 2015-10-20T13:49:01Z

@jfindlay i don't have a test case, because in my case on both our environments (stage and prod) the only needed step is to upgrade to 2015.5.5

tiger-seo · 2015-10-20T13:51:48Z

# salt --versions
           Salt: 2015.5.5
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.7.3
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 2.2.0.1
           RAET: Not Installed
            ZMQ: 3.2.4
           Mako: Not Installed
        Tornado: Not Installed
        timelib: Not Installed
       dateutil: Not Installed
# salt-call pillar.items
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
local:
    ----------

jfindlay · 2015-10-26T20:24:13Z

@tiger-seo, any more information you can provide about your pillar setup or salt setup that you think may be related to this would be helpful, thanks.

brucemartins · 2015-11-03T17:43:45Z

@jfindlay I see something similar and this is what the masters log outputs

2015-11-03 12:40:02,229 [salt.master                              ][ERROR   ][10272] Error in function _pillar:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1460, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1236, in _pillar
    'pillar': data})
  File "/usr/lib/python2.6/site-packages/salt/payload.py", line 122, in dumps
    except (OverflowError, msgpack.exceptions.PackValueError):
AttributeError: 'module' object has no attribute 'exceptions'

jfindlay · 2015-11-03T23:55:39Z

@brucemartins, this is a separate issue that has been fixed in #26714.

tiger-seo · 2015-11-04T07:21:16Z

@jfindlay please help me to provide you more information, because I have no idea

tiger-seo · 2015-11-04T07:22:26Z

Btw, supposedly, this error doesn't appear if I run state without pillar being used

jfindlay · 2015-11-11T21:34:57Z

@tiger-seo, have you investigated @sjwoodr suggestions?

tiger-seo · 2015-11-13T12:00:52Z

@jfindlay as i sad previously, the same problem appeared on both our environments, for sure it was not about disk free space or inodes ;)

kevins9 · 2015-12-16T22:40:44Z

I'm seeing this pop up on 2015.5.5 master/2015.5.5 minions if my pillar YAML contains a sequence of mappings. This is consistent and is affecting pillars that have been in use successfully for some time, which pass YAML lint. Here's an example:

nfsmounts:
- mountpoint: /foo/bar
  server: foobar01
  serverpath: /bar
- mountpoint: /foo/baz
  server: foobar01
  serverpath: /baz

UPDATE: Also affects a 2015.5.5 master/2015.8.1 minion combo.

cachedout · 2016-01-06T21:56:41Z

@kevins9 Using that pillar on a 2015.5 master/minion combo, I don't see any issues running salt my_minion pillar.items. Could you clarify under what conditions you see this?

mp@silver ...devel/salt/tests % sudo salt silver pillar.items                                              (git)-[v2015.5.5] 
silver:
    ----------
    nfsmounts:
        |_
          ----------
          mountpoint:
              /foo/bar
          server:
              foobar01
          serverpath:
              /bar
        |_
          ----------
          mountpoint:
              /foo/baz
          server:
              foobar01
          serverpath:
              /baz

kevins9 · 2016-01-06T22:16:17Z

Master:

[root@saltmaster ~]# salt --versions
           Salt: 2015.5.5
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
         Jinja2: 2.7.3
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 2.2.0.1
           RAET: Not Installed
            ZMQ: 3.2.4
           Mako: Not Installed
        Tornado: 2.2.1
        timelib: Not Installed
       dateutil: 1.4.1
[root@saltmaster pillar]# cat /srv/pillar/top.sls
base:
  'centos*':
    - pillartest


# vim: set ft=yaml ts=2 sw=2 et sts=2:
[root@saltmaster pillar]# cat /srv/pillar/pillartest.sls
nfsmounts:
- mountpoint: /foo/bar
  server: foobar01
  serverpath: /bar
- mountpoint: /foo/baz
  server: foobar01
  serverpath: /baz

Minion 1:

[root@centos7test-01 ~]# salt-minion --versions
Salt Version:
           Salt: 2015.8.1

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: 0.8.1
         PyYAML: 3.10
          PyZMQ: 14.3.1
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 3.2.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 3.10.0-229.11.1.el7.x86_64
         system: CentOS Linux 7.1.1503 Core

[root@centos7test-01 ~]# salt-call grains.get id
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
local:
    centos7test-01.foo.example.org

[root@centos7test-01 ~]# salt-call pillar.items
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
local:
    ----------

Minion 2:

[root@centos7test-02 ~]# salt-minion --versions
           Salt: 2015.5.5
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 3.2.5
           Mako: Not Installed
        Tornado: Not Installed
        timelib: Not Installed
       dateutil: Not Installed

[root@centos7test-02 ~]# salt-call grains.get id
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
local:
    centos7test-02.foo.example.org

[root@centos7test-02 ~]# salt-call pillar.items
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
[ERROR   ] Got a bad pillar from master, type str, expecting dict:
local:
    ----------

cachedout · 2016-01-06T22:22:57Z

Hmm, still not able to reproduce this. Anything in debug logs on master/minion?

kevins9 · 2016-01-06T22:42:39Z

On master, some instances of this in /var/log/salt/master:

2016-01-06 14:12:36,769 [salt.master                              ][ERROR   ][22404] Error in function _pillar:
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1460, in run_func
    ret = getattr(self, func)(load)
  File "/usr/lib/python2.6/site-packages/salt/master.py", line 1236, in _pillar
    'pillar': data})
  File "/usr/lib/python2.6/site-packages/salt/payload.py", line 122, in dumps
    except (OverflowError, msgpack.exceptions.PackValueError):
AttributeError: 'module' object has no attribute 'exceptions'

Appears relevant.

stk0vrfl0w · 2017-03-17T02:13:22Z

I'd noticed the same issue on our minions when we had the salt master cache pillar data to memory:

pillar_cache: True
pillar_cache_backend: memory

Using the default disk backend addressed the problem.

stale · 2018-09-18T03:00:03Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

jfindlay added the info-needed waiting for more info label Oct 9, 2015

jfindlay added this to the Blocked milestone Oct 9, 2015

jfindlay modified the milestones: Approved, Blocked Oct 19, 2015

jfindlay added Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around Core relates to code central or existential to Salt P4 Priority 4 and removed info-needed waiting for more info labels Oct 19, 2015

jfindlay mentioned this issue Oct 19, 2015

Salt 2015.5.2 - Could not deserialize msgpack message error. #25581

Closed

jfindlay added P3 Priority 3 Pillar Regression The issue is a bug that breaks functionality known to work in previous releases. and removed P4 Priority 4 labels Oct 26, 2015

kevins9 mentioned this issue Dec 16, 2015

state.sls fails to render state with pillar data: Jinja variable 'dict object' has no attribute #29028

Closed

vkurup mentioned this issue Jan 11, 2016

Upgrading salt on older projects causes failure caktus/django-project-template#237

Closed

Z9n2JktHlZDmlhSvqc9X2MmL3BwQG7tk mentioned this issue Jan 23, 2017

Root exec of salt-ssh breaks master #38886

Closed

stale bot added the stale label Sep 18, 2018

stale bot closed this as completed Sep 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ERROR ] Got a bad pillar from master, type str, expecting dict: #27792

[ERROR ] Got a bad pillar from master, type str, expecting dict: #27792

tiger-seo commented Oct 8, 2015

jfindlay commented Oct 9, 2015

sjwoodr commented Oct 19, 2015

sjwoodr commented Oct 19, 2015

jfindlay commented Oct 19, 2015

tiger-seo commented Oct 20, 2015

tiger-seo commented Oct 20, 2015

jfindlay commented Oct 26, 2015

brucemartins commented Nov 3, 2015

jfindlay commented Nov 3, 2015

tiger-seo commented Nov 4, 2015

tiger-seo commented Nov 4, 2015

jfindlay commented Nov 11, 2015

tiger-seo commented Nov 13, 2015

kevins9 commented Dec 16, 2015

cachedout commented Jan 6, 2016

kevins9 commented Jan 6, 2016

cachedout commented Jan 6, 2016

kevins9 commented Jan 6, 2016

stk0vrfl0w commented Mar 17, 2017

stale bot commented Sep 18, 2018

[ERROR ] Got a bad pillar from master, type str, expecting dict: #27792

[ERROR ] Got a bad pillar from master, type str, expecting dict: #27792

Comments

tiger-seo commented Oct 8, 2015

jfindlay commented Oct 9, 2015

sjwoodr commented Oct 19, 2015

sjwoodr commented Oct 19, 2015

jfindlay commented Oct 19, 2015

tiger-seo commented Oct 20, 2015

tiger-seo commented Oct 20, 2015

jfindlay commented Oct 26, 2015

brucemartins commented Nov 3, 2015

jfindlay commented Nov 3, 2015

tiger-seo commented Nov 4, 2015

tiger-seo commented Nov 4, 2015

jfindlay commented Nov 11, 2015

tiger-seo commented Nov 13, 2015

kevins9 commented Dec 16, 2015

cachedout commented Jan 6, 2016

kevins9 commented Jan 6, 2016

cachedout commented Jan 6, 2016

kevins9 commented Jan 6, 2016

stk0vrfl0w commented Mar 17, 2017

stale bot commented Sep 18, 2018