salt-master error on target VM when salt-cloud run as non-root

[alan-cugler]

Description of Issue

When provisioning salt-master onto a VM with salt-cloud, and salt-cloud is ran as an unprivileged (non-root) user, the salt-master fails on the target VM because an incorrect master configuration file is being produced by salt-cloud. This file among other things, sets user (on the target) as my username from the host computer I ran salt-cloud from. This results in salt-master (on the target) trying to set up the salt environment for the user alan-cugler. Again, to emphasize, alan-cugler doesn’t exist on the vm - it’s my username on the host I ran salt-cloud from.

It seems a different master.conf file is being created and used with the user: alan-cugler. I provide empty (except comments) salt/master and salt/cloud files.

Possibly related issue: #46904

I have included reference links for context, and steps to reproduce, all below:

Steps to Reproduce Issue

Setup

1. Downlaod bootstrap.sh
   wget -O bootstrap-salt.sh https://bootstrap.saltstack.com
2. Install salt
   sudo sh bootstrap-salt.sh -L -X -N -M
   NOTE:
   • -X Do not start daemons after installation
   • -N Do not install salt-minion
   • -L Also install salt-cloud
   • -M Also install salt-master
3. Use all relevant files shown below from the "configuration" section
4. Replace _syspaths.py default file with the one displayed in the "configuration" section
   cp -f _syspaths.py /usr/lib/python2.7/dist-packages/salt/_syspaths.py

Configuration

• /usr/lib/python2.7/dist-packages/salt/_syspaths.py

# Variables overlapping between auto-generation and online tutorial
ROOT_DIR = '/home/alan-cugler/testing/Salt_training/alans-salt-experiments'
CONFIG_DIR = ROOT_DIR + '/etc/salt'
CACHE_DIR = ROOT_DIR + '/var/cache/salt'
SOCK_DIR = ROOT_DIR + '/var/run/salt'
SRV_ROOT_DIR= ROOT_DIR + '/srv'
BASE_FILE_ROOTS_DIR = ROOT_DIR + '/srv/salt'
BASE_PILLAR_ROOTS_DIR = ROOT_DIR + '/srv/pillar'
BASE_MASTER_ROOTS_DIR = ROOT_DIR + '/srv/salt-master'
LOGS_DIR = ROOT_DIR + '/var/log/salt'
PIDFILE_DIR = ROOT_DIR + '/var/run'

# Variables not present in file but were in Docs
INSTALL_DIR = None
#CLOUD_DIR = INSTALL_DIR + '/cloud'
BOOTSTRAP = '/home/alan-cugler/testing/Salt_training/bootstrap-salt.sh'

# Variables present in file but not in Docs
SHARE_DIR = None
BASE_THORIUM_ROOTS_DIR = None
SPM_FORMULA_PATH = None
SPM_PILLAR_PATH = None
SPM_REACTOR_PATH = None
HOME_DIR = None

• [ ROOT_DIR from _syspaths.py ^ ]/etc/salt/cloud.providers.d/digitalocean-provider.conf

digitalocean:
  driver: digitalocean
  personal_access_token: [ Redacted ]
  ssh_key_file: [ Redacted ]
  ssh_key_names: [ Redacted ]

• [ ROOT_DIR from _syspaths.py ^ ]/etc/salt/cloud.profiles.d/digitalocean-profile.conf

do-ubuntu-1604-2gb-nyc1:
  provider: digitalocean
  image: 16.04.5 x64
  size: 2gb
  location: nyc1
  private_networking: True
  backups_enabled: False
  ipv6: False
  create_dns_record: False

• [ ROOT_DIR from _syspaths.py ^ ]/etc/salt/cloud.maps.d/digitalocean-map.conf

do-ubuntu-1604-2gb-nyc1:
  - master1:
      make_master: True
  - minion1
  - minion2

Error

Run salt-cloud with

salt-cloud -m cloud.maps.d/digitalocean-map.conf

This will result in a salt-master error on the Digitalocean VM.

When you login to the VM via ssh as root you can use

systemctl status salt-master -l

to see a similar error report:

● salt-master.service - The Salt Master Server
   Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2019-01-09 16:42:19 UTC; 1h 23min ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html
 Main PID: 10536 (code=exited, status=67)

Jan 09 16:42:18 master1 systemd[1]: Starting The Salt Master Server...
Jan 09 16:42:19 master1 salt-master[10536]: Failed to prepare the Salt environment for user alan-cugler. The user is not available.
Jan 09 16:42:19 master1 systemd[1]: salt-master.service: Main process exited, code=exited, status=67/n/a
Jan 09 16:42:19 master1 systemd[1]: Failed to start The Salt Master Server.
Jan 09 16:42:19 master1 systemd[1]: salt-master.service: Unit entered failed state.
Jan 09 16:42:19 master1 systemd[1]: salt-master.service: Failed with result 'exit-code'.

I have tried putting user: root in the salt/cloud, salt/master, and salt/cloud.maps.d/digitalocean-maps.conf files but this did not change anything besides my own system getting confused saying that salt-cloud had to be ran as the same user being listed in those files. In other words, I ran salt-cloud as a non-root user alan-cugler, which conflicts with root listed in those files.

Adding user:root to the salt/cloud file seems to apply to the master on my host, not the master on the target:

#salt/cloud
master:
  user: root

The docs show how to write variables to effect the minions, and I am guessing that I could do that for the master as well. If not I'd love to know how to.

Versions Report

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
   python-gnupg: 0.4.1
         PyYAML: 3.13
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5
 
System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-43-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Resources

• https://docs.saltstack.com/en/latest/topics/tutorials/rooted.html
• https://docs.saltstack.com/en/latest/ref/configuration/nonroot.html
• https://docs.saltstack.com/en/latest/topics/cloud/config.html#salt-cloud-config
• https://developers.digitalocean.com/documentation/v2/#droplets

[Ch3LL]

you can add master configuration options like so:

master:
  user: root
  interface: 0.0.0.0

in your profile or map file

https://docs.saltstack.com/en/latest/topics/cloud/misc.html#setting-up-new-salt-masters

[alan-cugler]

That worked great!
However, there was a new error to follow up this success. After some digging I will make a new issue for the error if I cant get it fixed.

Another problem is some possible indirect salt documentation conflicts. This doc says to use the master: syntax in the etc/salt/cloud file, but is this a special use case to the syndic_master: option only?

master:
  syndic_master: 123.456.789  # may be either an IP address or a hostname

I had assumed it extended in general to all master options.
https://docs.saltstack.com/en/latest/topics/cloud/misc.html#setting-up-a-salt-syndic-with-salt-cloud

[alan-cugler]

Also, this doc says the minion options are to be placed in the etc/salt/cloud file, without mention of the maps file. However, I just tested putting that option into the map file under the minion VM ( just like the master: option) and it worked!

So maybe a docs adjustment?
https://docs.saltstack.com/en/latest/topics/cloud/config.html#minion-configuration

[Ch3LL]

This doc says to use the master: syntax in the etc/salt/cloud file, but is this a special use case to the syndic_master: option only?

it should encompass all master options if you set them in the map file

Also, this doc says the minion options are to be placed in the etc/salt/cloud file, without mention of the maps file.

you can set them in either the cloud or map file. since those docs are specific to /etc/salt/cloud and map has its own documentation i think this is clear enough unless you see otherwise

[alan-cugler]

I disagree on the docs handling. If Saltstack is formatting their docs to keep each configuration file separate, then their needs to be redundancy to re-explain how that same variable/setting/flag should be formatted and used in the new configuration file.

But that is exactly what is not happening here. All over the salt docs they mention a variable/setting/flag, show an example of it, and then list the files it can be placed in.

Therefore, the doc that says the configuration variable can explicitly be set in the cloud file needs to be changed to state it can be done in the cloud file or the map file.

[alan-cugler]

I am happy to do it myself. Though, it may take time since I haven't done so before, and I am a bit slow on getting through all your procedure docs for contributing.

[Ch3LL]

I agree we could make it more clear on each doc to make sure its clear this option is able to be set elsewhere :)

For contributing since this is documentation you probably want to submit it to the 2018.3 branch so it shows up on that branch and 2019.2, and develop once its merged forward.

The docs for submitting documentation is here: https://docs.saltstack.com/en/latest/topics/development/conventions/documentation.html

let me know if you have any other questions. I would be glad to help

[alan-cugler]

Alright, I am trying to make the documentation changes for this. However, after building the docs and making the changes to the relevant html page I realized the docs/_build/ is in .gitignore.
So, am I suppose to erase this in the .gitignore so that I can commit it and PR from my forked repo?

[alan-cugler]

https://docs.saltstack.com/en/latest/topics/development/hacking.html#editing-and-previewing-the-documentation
I found this, but the "run make again" doesnt seem to cause git to see the changes

#51696

[Ch3LL]

you wont want to submit the changes that are added to the _build folder. Can you clarify the exact files that you are trying to edit?

[alan-cugler]

@Ch3LL this issue is marked with the bug label, and I do think there is a bug.
However, I intended to mark this closed after the doc change PR is accepted.

Do you want me to make a clone issue for the bug or leave this one open?

[Ch3LL]

i labeled this a documentation bug, not a bug in salt's code. What issue are you referring to? And yes if there is another issue outside of documentation lets close here and open a new one

[alan-cugler]

The original error this issue was based off of had peculiar behavior. Before you showed me how to set the host as the deescalated user properly, every time I created a new vm from salt-cloud that vm would try to do everything in my deescalated user.

The behavior that I thought was a bug, was "an incorrect master configuration file is being produced by salt-cloud."
This master config file is generated some how by salt, it looks nothing like the normal config file.

[alan-cugler]

I am going to close this issue and if someone runs into the original error and odd behavior they can re-open this for further discussion.