Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Using sdb.get from Vault in salt-cloud or salt-master config breaks Vault integration #59846

Open
Arderos opened this issue Mar 20, 2021 · 11 comments
Labels
Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Milestone

Comments

@Arderos
Copy link

Arderos commented Mar 20, 2021

Description
If salt is configured with Hashicorp Vault integration via approle and 'sdb.get' is used in salt or salt-cloud configs an empty "salt_vault_token" file is placed in /var/cache/salt/master/ or /var/cache/salt/cloud/ respectively. With this file present, sdb.get runner stops returning information from Vault leading to salt-master start failure of salt-cloud failure to work with a provider in which the 'sdb.get' option is used.

Setup
/etc/salt/master.d/vault.conf:

# SDB CONFIG
myvault:
  driver: vault

# VAULT CONFIG FOR TOKEN OR APPROLE AUTH MODE
vault:
  url: https://[REDACTED]:8200
  verify: /etc/ssl/certs/ca-certificates.crt
  auth:
    method: approle
    role_id: [REDACTED]
    secret_id: [REDACTED]
  policies:
    - saltmaster
  keys:
    - [REDACTED]

/etc/salt/cloud.providers.d/vcenter.conf:

vcenter:
  driver: vmware
  user: '[REDACTED]'
  password: 'sdb://myvault/salt/vsphere/[REDACTED]'
  url: '[REDACTED]'
  protocol: 'https'
  port: 443

/etc/salt/master.d/rest.conf:

rest_cherrypy:
  port: 8000
  ssl_crt: 'sdb://myvault/salt/certificates/salt_rest_cert'
  ssl_key: 'sdb://myvault/salt/certificates/salt_rest_key'

Steps to Reproduce the behavior
Install salt, salt-cloud and salt-api on a server
Configure vault integration via approle
Put a pointer to a secret in salt config file
Restart the salt-master.
Salt debug log:

[DEBUG   ] LazyLoaded vault.get
[DEBUG   ] LazyLoaded vault.is_v2
[ERROR   ] module 'salt.utils' has no attribute 'json'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 196, in parse_args
    process_option_func()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 553, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1948, in setup_config
    opts = config.master_config(self.get_config_file_path())
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3789, in master_config
    apply_sdb(opts)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2379, in apply_sdb
    return salt.utils.sdb.sdb_get(sdb_opts, opts)
  File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 60, in sdb_get
    return loaded_db[fun](query, profile=profile)
  File "/usr/lib/python3/dist-packages/salt/sdb/vault.py", line 89, in get
    version2 = __utils__["vault.is_v2"](path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 406, in is_v2
    path_metadata = _get_secret_path_metadata(path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 473, in _get_secret_path_metadata
    cache_content = _read_cache_file()
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 244, in _read_cache_file
    return salt.utils.json.load(contents)
AttributeError: module 'salt.utils' has no attribute 'json'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
TypeError: '>=' not supported between instances of 'AttributeError' and 'int'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 196, in parse_args
    process_option_func()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 553, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1948, in setup_config
    opts = config.master_config(self.get_config_file_path())
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3789, in master_config
    apply_sdb(opts)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2379, in apply_sdb
    return salt.utils.sdb.sdb_get(sdb_opts, opts)
  File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 60, in sdb_get
    return loaded_db[fun](query, profile=profile)
  File "/usr/lib/python3/dist-packages/salt/sdb/vault.py", line 89, in get
    version2 = __utils__["vault.is_v2"](path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 406, in is_v2
    path_metadata = _get_secret_path_metadata(path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 473, in _get_secret_path_metadata
    cache_content = _read_cache_file()
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 244, in _read_cache_file
    return salt.utils.json.load(contents)
AttributeError: module 'salt.utils' has no attribute 'json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt-master", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-master')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 104, in salt_master
    master.start()
  File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 213, in start
    super(Master, self).start()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1095, in start
    self.prepare()
  File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 148, in prepare
    super(Master, self).prepare()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1092, in prepare
    self.parse_args()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 201, in parse_args
    process_option_func, traceback.format_exc(err)
  File "/usr/lib/python3.8/traceback.py", line 167, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/usr/lib/python3.8/traceback.py", line 120, in format_exception
    return list(TracebackException(
  File "/usr/lib/python3.8/traceback.py", line 509, in __init__
    self.stack = StackSummary.extract(
  File "/usr/lib/python3.8/traceback.py", line 340, in extract
    if limit >= 0:
TypeError: '>=' not supported between instances of 'AttributeError' and 'int'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 196, in parse_args
    process_option_func()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 553, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1948, in setup_config
    opts = config.master_config(self.get_config_file_path())
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3789, in master_config
    apply_sdb(opts)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2384, in apply_sdb
    sdb_opts[key] = apply_sdb(opts, value)
  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 2379, in apply_sdb
    return salt.utils.sdb.sdb_get(sdb_opts, opts)
  File "/usr/lib/python3/dist-packages/salt/utils/sdb.py", line 60, in sdb_get
    return loaded_db[fun](query, profile=profile)
  File "/usr/lib/python3/dist-packages/salt/sdb/vault.py", line 89, in get
    version2 = __utils__["vault.is_v2"](path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 406, in is_v2
    path_metadata = _get_secret_path_metadata(path)
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 473, in _get_secret_path_metadata
    cache_content = _read_cache_file()
  File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 244, in _read_cache_file
    return salt.utils.json.load(contents)
AttributeError: module 'salt.utils' has no attribute 'json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/salt-master", line 11, in <module>
    load_entry_point('salt==3002.5', 'console_scripts', 'salt-master')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 104, in salt_master
    master.start()
  File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 213, in start
    super(Master, self).start()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1095, in start
    self.prepare()
  File "/usr/lib/python3/dist-packages/salt/cli/daemons.py", line 148, in prepare
    super(Master, self).prepare()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 1092, in prepare
    self.parse_args()
  File "/usr/lib/python3/dist-packages/salt/utils/parsers.py", line 201, in parse_args
    process_option_func, traceback.format_exc(err)
  File "/usr/lib/python3.8/traceback.py", line 167, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/usr/lib/python3.8/traceback.py", line 120, in format_exception
    return list(TracebackException(
  File "/usr/lib/python3.8/traceback.py", line 509, in __init__
    self.stack = StackSummary.extract(
  File "/usr/lib/python3.8/traceback.py", line 340, in extract
    if limit >= 0:
TypeError: '>=' not supported between instances of 'AttributeError' and 'int'

Debug log from salt-cloud is the same.
Expected behavior
Salt and salt-cloud should get the data from Vault and use it to fill the relevant configuration variables.

Screenshots
Not applicable

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3002.5
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: 4.1.0
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.31.0
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2
 
System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-67-generic
        system: Linux
       version: Ubuntu 20.04 focal

Additional context
Add any other context about the problem here.

@Arderos Arderos added Bug broken, incorrect, or confusing behavior needs-triage labels Mar 20, 2021
@sagetherage
Copy link
Contributor

@garethgreenaway similar to #58897?

@sagetherage sagetherage added severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around and removed needs-triage labels Apr 30, 2021
@sagetherage sagetherage added this to the Approved milestone Apr 30, 2021
Copy link
Contributor

Looks like something different.
@Arderos does /usr/lib/python3/dist-packages/salt/utils/json.py exist?

@Arderos
Copy link
Author

Arderos commented May 25, 2021

Looks like something different.
@Arderos does /usr/lib/python3/dist-packages/salt/utils/json.py exist?

It does.

@JeremyHutchings
Copy link

JeremyHutchings commented Jun 28, 2021

@Arderos when you say :

Configure vault integration via approle

What exact steps are you taking there, as the only output I'm getting at the moment is :

[ERROR ] Failed to get token from master! An error was returned: Bad Request
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/modules/vault.py", line 366, in list_secrets
response = utils["vault.make_request"]("LIST", url)
File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 298, in make_request
connection = get_cache()
File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 263, in get_cache
return _gen_new_connection()
File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 256, in _gen_new_connection
connection = get_vault_connection()
File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 179, in get_vault_connection
return _get_token_and_url_from_master()
File "/usr/lib/python3/dist-packages/salt/utils/vault.py", line 97, in _get_token_and_url_from_master
raise salt.exceptions.CommandExecutionError(result)
salt.exceptions.CommandExecutionError: {'error': 'Bad Request'}

And there is no logging or indication of what is going on.

@Arderos
Copy link
Author

Arderos commented Jul 6, 2021

What exact steps are you taking there, as the only output I'm getting at the moment is :

The configuration for approle is in the issue description, in /etc/salt/master.d/vault.conf file.
Vault configuration:
http://web.archive.org/web/20201205000303/https://help.saltstack.com/hc/en-us/articles/360041140451-Quick-Guide-to-Vault-Integration with only difference being that I do not use V2 engine.

@JeremyHutchings
Copy link

Turns out my issues is a known bug for how salt interacts with Vault enterprise (no passing of namespace) which is fixed and waiting for release in a future version.

@Arderos
Copy link
Author

Arderos commented Dec 10, 2021

@JeremyHutchings is there any roadmap for this release you mentioned?

@cathode911
Copy link

Turns out my issues is a known bug for how salt interacts with Vault enterprise (no passing of namespace) which is fixed and waiting for release in a future version.

Jeremy, can you please elaborate when this fix will be merged? It's still reproducing in version 3004 and basically is a blocker for us to use salt-cloud - corporate policy does not allow to store any kind of passwords in plain text

@JeremyHutchings
Copy link

I think this might be it : #58586 I don't know about which release it was supposed to be in

@Arderos
Copy link
Author

Arderos commented Dec 10, 2021

I think this might be it : #58586 I don't know about which release it was supposed to be in

I have tried 3004+0na.042b788 from git. Same error.

@Arderos
Copy link
Author

Arderos commented May 18, 2022

Guys, any progress?

lkubb added a commit to lkubb/salt-vault-formula that referenced this issue Oct 5, 2022
This commit represents a fundamental rewrite in how Salt interacts with
Vault. The master should still be compatible with minions running the
old code. There should be no breaking changes to public interfaces and
the old configuration format should still apply.

Core:
- Issue AppRoles to minions
- Manage entities with templatable metadata for minions
- Use inbuilt Salt cache
- Separate config cache from token cache
- Cache: introduce connection-scope vs global scope

Utility module:
- Support being imported (__utils__ deprecation)
- Raise exceptions on queries to simplify response handling
- Add classes to wrap complexity, especially regarding KV v2
- Lay some groundwork for renewing tokens

Execution module:
- Add patch_secret
- Add version support to delete_secret
- Allow returning listed keys only in list_secret
- Add policy_[fetch/write/delete] and policies_list
- Add query for arbitrary API queries

State module:
- Make use of execution module
- Change output format

Docs:
- Update for new configuration format
- Correct examples
- Add configuration examples
- Add required policies

Fixes:
saltstack/salt#62552
saltstack/salt#59827
saltstack/salt#62380
saltstack/salt#58174

Probably fixes:
saltstack/salt#60779
saltstack/salt#57561

Might fix:
saltstack/salt#59846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior severity-medium 3rd level, incorrect or bad functionality, confusing and lacks a work around
Projects
None yet
Development

No branches or pull requests

5 participants