[BUG] Sending messages to Slack when behind a proxy fails with HTTP 400 Bad Request response.

[justinclloyd]

Description
Sending messages to a Slack channel from behind a proxy results in a "400 Bad Request" response from hooks.slack.com.

Setup
This can be reproduced with a simple SLS file, ensuring that appropriate channel and webhook values are specified:

send_slack_message:
  slack.post_message:
    - message: "slack.post_message test"
    - channel: "#some_channel_name"
    - webhook: "https://hooks.slack.com/services/REDACTED"

Any appropriate salt-master settings for proxy_host and proxy_port should reproduce the problem. In my case, I'm testing on an AWS EC2 instance so that I can enable and disable a Squid service running on localhost:3128 by simply removing or commenting out those settings and restarting the salt-master service.

• on-prem machine: Initially discovered issue on an on-prem Hyper-V Salt master.
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service: Reproduced on AWS EC2
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging (3004.2)
• onedir packaging (3006.7)
• used bootstrap to install

Steps to Reproduce the behavior

1. Enable salt-master proxy_host and proxy_port settings to point at your proxy. We use Squid.
2. Put the above send_slack_message in an SLS file, e.g. slacktest.sls
3. salt-call state.apply slacktest

Relevant info from /var/log/salt/master:

2024-02-22 12:26:43,909 [salt.utils.http  ][   244][DEBUG   ][61223] Switching to request backend due to the use of proxies.
2024-02-22 12:26:43,910 [salt.utils.http  ][   292][DEBUG   ][61223] Requesting URL https://hooks.slack.com/services/REDACTED using POST method
2024-02-22 12:26:43,910 [salt.utils.http  ][   293][DEBUG   ][61223] Using backend: requests
2024-02-22 12:26:43,911 [salt.utils.http  ][    32][TRACE   ][61223] Request POST Data: 'payload=%7B%22text%22%3A+%22Minion+use1a-00-6a7f.example.com+started+at+Thu+Feb+22+12%3A26%3A41+2024%22%2C+%22channel%22%3A+%22%23gameops_salt_events%22%7D'
2024-02-22 12:26:43,911 [salt.utils.http  ][    32][TRACE   ][61223] Request Headers: {'User-agent': 'Salt/3006.7 http.query()', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
2024-02-22 12:26:43,913 [urllib3.connectionpool][  1019][DEBUG   ][61223] Starting new HTTPS connection (1): hooks.slack.com:443
2024-02-22 12:26:44,038 [urllib3.connectionpool][   474][DEBUG   ][61223] https://hooks.slack.com:443 "POST /services/REDACTED HTTP/1.1" 400 15
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2424, in call
    ret = self.states[cdata["full"]](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1293, in wrapper
    return f(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/slack.py", line 153, in post_message
    result = __salt__["slack.call_hook"](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/slack_notify.py", line 312, in call_hook
    result = salt.utils.http.query(url, method="POST", data=data, status=True)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/http.py", line 427, in query
    result.raise_for_status()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://hooks.slack.com/services/REDACTED

Expected behavior
The message value should have been posted to the specified channel at the webhook URL. It was not due to the HTTP 400 response.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.7
 
Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:34:22) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: aarch64
       release: 6.2.0-1018-aws
        system: Linux
       version: Ubuntu 22.04.2 jammy```

Additional context
From my troubleshooting, the issue seems to be due to call_hook in slack_notify.py always sending the data as a URL-encoded string rather than as JSON, which is required when using the requests library. When not using a proxy, Salt is defaulting to use tornado, for which the string sintead of a dict seems to work fine, meaning Squid (and possibly other proxies) don't like the POST data not being JSON.

data = urllib.parse.urlencode({"payload": salt.utils.json.dumps(payload)})
result = salt.utils.http.query(url, method="POST", data=data, status=True)

[dwoz]

Based on conversations in the community slack thread:

• Peviously the curl backend would assume application/x-www-form-urlencoded
• Until the bug is fixed a simple work around is the following patch:

diff --git a/salt/modules/slack_notify.py b/salt/modules/slack_notify.py
index d0d32fabe18..39b1e715933 100644
--- a/salt/modules/slack_notify.py
+++ b/salt/modules/slack_notify.py
@@ -309,7 +309,8 @@ def call_hook(
         payload["icon_emoji"] = icon_emoji
 
     data = urllib.parse.urlencode({"payload": salt.utils.json.dumps(payload)})
-    result = salt.utils.http.query(url, method="POST", data=data, status=True)
+    header_dict = {"Content-Type": "application/x-www-form-urlencoded"}
+    result = salt.utils.http.query(url, method="POST", data=data, status=True, header_dict=header_dict)
 
     if result["status"] <= 201:
         return True

[dwoz]

Fixed in 3006.8