2016.3.1 broke pillar merging

[mohshami]

Description of Issue/Question

Hi guys,

I have recently upgraded from 2015.8.7 to 2016.3.1 and it broke my orchestration. I have set up my environment with each web server having a pillar file for each website that it has

Setup

Relivant part of the pillar file

pillar/websites/site1.sls
websites:
  - site1

pillar/websites/site2.sls
websites:
  - site2

Steps to Reproduce Issue

With 2015.8.7, running salt-call pillar.items websites returns an array of site1 and site2, now the same command only includes that last defined website

From master: pillar_source_merging_strategy: recurse

Then I have changed pillar_merge_lists to True, so now running pillar.items website returns both websites in the array, but this causes a second issue

I have an orchestrator that runs like below

/usr/local/bin/salt-run state.orchestrate deploy pillar='{"websites": ["site1"]}'

This way I can deploy multiple sites at once, when I run it I get

master_master:
    Data failed to compile:
----------
    Rendering SLS 'base:deploy' failed: Conflicting ID 'site1_excludeFiles'

The sls file is as below

{% for website in pillar.get('websites') %}

# Generate the list of files to be excluded from the code deployment
{{ website }}_excludeFiles:
  salt.state:
    - tgt: '{{ pillar.get(website)['target'] }}'
    - sls: deploy.excludeFiles
      pillar:
         website: {{ pillar.get(website) }}
         project: {{ salt['pillar.get'](salt['pillar.get'](website)['code']) }}

{{ website }}_gitDeploy:
  salt.state:
    - tgt: '{{ pillar.get(website)['target'] }}'
    - sls: deploy.git
      pillar:
         website: {{ pillar.get(website) }}
    - require:
      - salt: {{ website }}_excludeFiles

{{ website }}_setPermissions:
  salt.state:
    - tgt: '{{ pillar.get(website)['target'] }}'
    - sls: deploy.permissions
      pillar:
         website: {{ pillar.get(website) }}
         project: {{ salt['pillar.get'](salt['pillar.get'](website)['code']) }}
    - require:
      - salt: {{ website }}_gitDeploy
{% endfor %}

What I have just found is that pillar='{"websites": ["site1"]}' in the command above used to overwrite the websites pillar and allow me to deploy without issues, but now it merges site1 with the current websites pillar so now I have a duplicate site1

Is there a way to overwrite pillars when passing through the command line and merging them otherwise?

Versions Report

Same for master and minion
Salt Version:
           Salt: 2016.3.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jul 18 2016, 07:20:22)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.4

System Versions:
           dist:
        machine: amd64
        release: 10.2-RELEASE
         system: FreeBSD
        version: Not Installed

[Ch3LL]

@mohshami just to clarify it seems the issue now is that the pillar you are passing via command line is now also being merged into your other pillar values which causes the ID conflict. Am I understanding that correctly?

If that is the issue it looks like this behavior as been changed in #32288 to fix the issue #18429 to ensure pillar data on the command line uses merge instead of update to mirror the merge settings in your configs. So it appears this is now expected bheavior, but I think we could document this better so I will label this as a documentation bug. Please let me know if i understood this issue correctly thanks

[mohshami]

Thank you Megan,

Yes that is part of the issue, so i changed the pillar name, but another issue is that a loop I have generates duplicate entries. I changed 'websites' to 'websites2deploy'

{% for website in pillar.get('websites2deploy') %}

Generate the list of files to be excluded from the code deployment

{{ website }}_excludeFiles:
salt.state:
- tgt: '{{ pillar.get(website)['target'] }}'
- sls: deploy.excludeFiles
pillar:
website: {{ pillar.get(website) }}
project: {{ salt'pillar.get' }}

{{ website }}_gitDeploy:
salt.state:
- tgt: '{{ pillar.get(website)['target'] }}'
- sls: deploy.git
pillar:
website: {{ pillar.get(website) }}
- require:
- salt: {{ website }}_excludeFiles

{{ website }}_setPermissions:
salt.state:
- tgt: '{{ pillar.get(website)['target'] }}'
- sls: deploy.permissions
pillar:
website: {{ pillar.get(website) }}
project: {{ salt'pillar.get' }}
- require:
- salt: {{ website }}_gitDeploy
{% endfor %}

Even though {{ website }}_excludeFiles should be unique and it worked on the previous version, now I get an error saying it's duplicated

Thanks again for your help

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.