[BUG] salt-api 401 Unauthorized

[boozy1337]

Description
Since upgrading two of our Saltstack environments we seem to face a bug of the salt-api that refuses any requests.
The problem occurs on both systems, although I am only giving details of one environment so far. But both are pretty similar.
The API worked like expected before.

As you can see I am running the OneDir version 3006.4 as root. Previously we were running version 3006.3 and had a lot of problems with the NonceVerificationError

The login function works well and returns a token and also shows the right permissions.
curl -k http://salt:8080/login -H "Accept:application/json" -H "Content-type:application/json" -d '{"username":"tux", "password":"areallystrongpassword", "eauth":"pam"}' {"return": [{"token": "areallysecuretoken", "expire": 1700522599.8706264, "start": 1700479399.8706262, "user": "tux", "eauth": "pam", "perms": [".*", "@jobs", "@runner", "@wheel"]}]}

But the following request fails.
curl -k http://salt:8080/ -H "Content-type:application/json" -H "X-Auth-Token:areallysecuretoken" -d '[{"client":"wheel","fun":"key.list"}]'
No matter what fun and client I use and also when using "eauth=pam" with the same credentials. The result is always the same:

<h2>401 Unauthorized</h2>
        <p>No permission -- see authorization schemes</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 688, in _do_respond
    self.hooks.run('before_request_body')
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 95, in run
    self.run_hooks(iter(sorted(self[point])))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 117, in run_hooks
    hook()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/cherrypy/_cprequest.py", line 65, in __call__
    return self.callback(**self.kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/netapi/rest_cherrypy/app.py", line 781, in salt_auth_tool
    raise cherrypy.HTTPError(401)
cherrypy._cperror.HTTPError: (401, None)
</pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 18.6.1</a>
      </span>
    </div>

Here's what's in the log:

Nov 20 12:31:34 salt salt-api[23062]: 10.8.25.129 - - [20/Nov/2023:12:31:34] "POST /login HTTP/1.1" 200 208 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Nov 20 12:31:48 salt salt-master[22446]: [WARNING ] Authentication failure of type "token" occurred.
Nov 20 12:31:48 salt salt-api[23062]: [ERROR   ] Authorization error occurred.
Nov 20 12:31:48 salt salt-api[23062]: 10.8.25.129 - - [20/Nov/2023:12:31:48] "POST / HTTP/1.1" 401 1036 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"

Seems to me like somehow the master is the problem here that refuses the login.

On the CLI the user is able to run the commands:

root@salt:/etc/salt# salt -a pam --username=tux sdev-salt01 test.ping
password:
sdev-salt01:
    True

Hopefully I am not wasting your time with a pretty stupid layer 8 problem.
Thanks a lot!

ADDITION:
I am still looking for a solution and found out a few additional things:

• The following request runs as part of on of our gitlab pipelines and it still works:
  curl -sS http://$SALTAPI/hook/$CI_PROJECT_NAME/commit -H "Content-type:application/json" -H "X-Auth-Token:$TOKEN" -d '{"user-mail":"$GITLAB_USER_EMAIL", "user-name":"$GITLAB_USER_NAME"}' | grep success
• This request works too:
  curl http://salt02:8080/run -H 'Accept: application/x-yaml' -d username=tux -d password=securepassword -d eauth=pam -d client=local -d tgt='*' -d fun=test.ping

So, right now it looks to me like there is a problem with all the requests on the root path of the webserver.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (VMware)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

salt-api config:

rest_cherrypy:
  host: 0.0.0.0
  port: 8080
  debug: True
  disable_ssl: True
  log_access_file: /var/log/salt/api-access.log
  log_error_file: /var/log/salt/api-error.log

netapi_enable_clients:
  - local
  - local_async
  - local_batch
  - local_subset
  - runner
  - runner_async
  - wheel
  - wheel_async

external_auth:
  pam:
    tux:
      - '.*'
      - '@runner'
      - '@jobs'
      - '@wheel'

Master Config:

root@salt:/opt# cat /etc/salt/master | grep -vE '^#' | grep -vE '^$'
user: root
event_return: mysql
keep_acl_in_token: True

Here are some more logs of the master on'debug' level:

Nov 20 13:06:19 salt salt-master[46022]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:23 salt salt-master[46014]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:26 salt salt-master[46025]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:27 salt salt-master[46024]: [DEBUG   ] salt.crypt.get_rsa_pub_key: Loading public key
Nov 20 13:06:28 salt salt-master[46003]: [WARNING ] Authentication failure of type "token" occurred.
Nov 20 13:06:28 salt salt-master[46003]: [DEBUG   ] Sending event: tag = salt/job/error; data = {'cmd': 'publish', 'tgt': 'sdev-salt02', 'fun': 'test.ping', 'arg': [], 'key': 'somekey', 'tgt_type': 'glob', 'ret': '', 'jid': '', 'kwargs': {'client': 'local', 'token': 'reallysecuretoken'}, 'user': 'root', 'error': {'name': 'AuthorizationError', 'message': 'Authorization error occurred.'}, '_stamp': '2023-11-20T12:06:28.238982'}
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Flushing 1 events.
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Calling event returner mysql, only one configured.
Nov 20 13:06:28 salt salt-master[45958]: [DEBUG   ] Gathering reactors for tag salt/job/error
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_ca
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_cert
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Using default for mysql ssl_key
Nov 20 13:06:28 salt salt-master[45997]: [DEBUG   ] Trying to reuse MySQL connection pool
Nov 20 13:06:28 salt salt-api[36960]: [ERROR   ] Authorization error occurred.
Nov 20 13:06:28 salt salt-api[36960]: 10.8.25.129 - - [20/Nov/2023:13:06:28] "POST / HTTP/1.1" 401 1036 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"

Steps to Reproduce the behavior

Expected behavior
Please just do what I want :)

Screenshots

Versions Report

salt --versions-report

Salt Version: Salt: 3006.4

Python Version:
Python: 3.10.13 (main, Oct 4 2023, 21:54:22) [GCC 11.2.0]

Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: 1.7.1
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.13.1
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.13.12
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4

System Versions:
dist: ubuntu 22.04.3 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-88-generic
system: Linux
version: Ubuntu 22.04.3 jammy

Additional context

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[bergentroll]

I have now looking the same problem while keep_acl_in_token: True in combination with auth.file. Token header works good when this line is commented out.

[bergentroll]

It looks like keep_acl_in_token: True gives the vague error with auth modules which do not implement acl() function (like auth and file, and seems pam also).

[boozy1337]

First of all thanks for your further investigation of the problem.

It doesn't only impair pam. rest is also affected.
I tried it with keep_acl_in_token: False and it works.
I am a bit confused that nobody else seems to use this option ( that is forced by rest auth ).