Discord messages not correctly formatted/delivered

[dnnnvx]

What did you do?

• Set up kube-stack on a k8s cluster (Alertmanager deployed via Prometheus Operator)
• Configured Alertmanager for discord notifications following the official blog article guide

What did you expect to see?
Discord notifications correctly delivered

What did you see instead? Under which circumstances?
Alertmanager cannot deliver the message, prints the error log:

ts=2023-10-30T16:27:09.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="discord/discord[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: {\"embeds\": [\"0\"]}"

Such response from Discord means that some Embeds are bad configured, so I guess the JSON payload created is not well formatted. Couldn't find a way to debug the created payload (via logLevel=debug)

Here's the simple alert I'm testing which always trigger the alert (as stated via the UI too):

- alert: TestAlert
  expr: |
    test_metrics_count > 1
  for: 1m
  labels:
    severity: critical
  annotations:
    title: "Test Alert"
    description: "Test Alert"

Environment

• System information:
  Linux 5.15.117 x86_64

• Alertmanager version:

alertmanager, version 0.26.0 (branch: HEAD, revision: d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)
  build user:       root@df8d7debeef4
  build date:       20230824-11:11:58
  go version:       go1.20.7
  platform:         linux/amd64
  tags:             netgo

• Prometheus version:

prometheus, version 2.47.1 (branch: HEAD, revision: c4d1a8beff37cc004f1dc4ab9d2e73193f51aaeb)
  build user:       root@4829330363be
  build date:       20231004-10:31:16
  go version:       go1.21.1
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

• Alertmanager configuration file:

global:
  resolve_timeout: 5m
inhibit_rules:
- equal:
  - namespace
  - alertname
  source_matchers:
  - severity = critical
  target_matchers:
  - severity =~ warning|info
- equal:
  - namespace
  - alertname
  source_matchers:
  - severity = warning
  target_matchers:
  - severity = info
- equal:
  - namespace
  source_matchers:
  - alertname = InfoInhibitor
  target_matchers:
  - severity = info
receivers:
- name: "null"
- discord_configs:
  - send_resolved: true
    webhook_url: https://discord.com/api/webhooks/.../...
  name: discord
route:
  group_by:
  - alertname
  - job
  group_interval: 5m
  group_wait: 30s
  receiver: discord
  repeat_interval: 12h
  routes:
  - matchers:
    - alertname =~ "InfoInhibitor|Watchdog"
    receiver: "null"
  - matchers:
    - severity =~ "critical|error|warning|info"
    receiver: discord
templates:
- /etc/alertmanager/config/*.tmpl

• Prometheus configuration file:
  Not relevant

• Logs:

ts=2023-10-30T16:27:09.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="discord/discord[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: {\"embeds\": [\"0\"]}"

[dnnnvx]

According to this that Discord error message pops up on a various type of format errors, the most frequent one is an empty embed field. I am now trying to do a bit of reverse engineering and modifying a the template to see if that's the case

[dnnnvx]

After some changes in the template, I was able to make it works, so I can confirm it's due to some empty fields in the embeds

title: >-
  [{{ .Status | toUpper }}
  {{ if eq .Status "firing" }}: {{ .Alerts.Firing | len }}{{ end }}]
  {{ with (index .GroupLabels.SortedPairs.Values 0) }}
  {{ . }}
  {{ end }}
message: |-
  {{ if and .Alerts.Firing (gt (len .Alerts.Firing) 0) }}
  {{- range .Alerts.Firing }}
  {{- if gt (len .Annotations) 0 }}
  {{ range .Annotations.SortedPairs }}
  - {{ .Name }}: {{ .Value }}
  {{- end }}
  {{- else }}
  (No Annotations)
  {{- end }}
  {{- end }}
  {{- end }}

Obviously this is a slim version (I don't need all of the infos) but my suggestion was to check for the fields in the code and not in the template and maybe add a default string - for missing values

[Vinaum8]

After some changes in the template, I was able to make it works, so I can confirm it's due to some empty fields in the embeds

title: >-
  [{{ .Status | toUpper }}
  {{ if eq .Status "firing" }}: {{ .Alerts.Firing | len }}{{ end }}]
  {{ with (index .GroupLabels.SortedPairs.Values 0) }}
  {{ . }}
  {{ end }}
message: |-
  {{ if and .Alerts.Firing (gt (len .Alerts.Firing) 0) }}
  {{- range .Alerts.Firing }}
  {{- if gt (len .Annotations) 0 }}
  {{ range .Annotations.SortedPairs }}
  - {{ .Name }}: {{ .Value }}
  {{- end }}
  {{- else }}
  (No Annotations)
  {{- end }}
  {{- end }}
  {{- end }}

Obviously this is a slim version (I don't need all of the infos) but my suggestion was to check for the fields in the code and not in the template and maybe add a default string - for missing values

@dnnnvx Where should I put the settings to use with discord?

[dnnnvx]

@dnnnvx Where should I put the settings to use with discord?

(Note that I'm using the kube-prom-stack chart) in the alertmanager values you need to add a discord receiver (I ended up using the slack title/text templates):

alertmanager:
  ...
  receivers:
    ...
    - name: discord
      discord_configs:
        - webhook_url: "https://discord.com/api/webhooks/..."
          send_resolved: true
          title: '{{ template "slack.title" . }}'
          message: '{{ template "slack.text" . }}'

[bmassemin]

I ran into the same issue, and I think it is related to missingkey=zero option in the template parser.
So each field should be tested before being used.

[uhthomas]

Duplicate of #3310 and fixed by #3597? Currently unreleased, though.

[dnnnvx]

Duplicate of #3310 and fixed by #3597? Currently unreleased, though.

I don't think it's the same, #3310 is about limits, here's the issue is about missing proper fields checks, so each field needs to be manually checked (while that's not the case with, for example, Slack)

[uhthomas]

My understanding is the error message is identical.

ts=2023-10-30T16:27:09.440Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="discord/discord[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: {\"embeds\": [\"0\"]}"

How can we be sure it's not the same? I recognise there has been some triaging here, and changing the template has worked for you. It does confuse me however, as the actual Discord integration works okay.

Could you verify you can still reproduce the issue with quay.io/prometheus/alertmanager:main@sha256:889198464d4f78f7b0781d5ab76fa7c97a054dc3ba0f00395b1a4efdb9f7edd4?

https://quay.io/repository/prometheus/alertmanager/manifest/sha256:889198464d4f78f7b0781d5ab76fa7c97a054dc3ba0f00395b1a4efdb9f7edd4

[dnnnvx]

Sorry @uhthomas you're right, I was looking at the wrong log- I deployed alertmanager in my cluster with that SHA and it seems to work fine, no error for now. The error I found (the reason of this PR) was not for each alert, so I'll continue to keep an eye on it. For now, seems to be solved with that version!

[dnnnvx]

Received 2 new errors tonight and both were displayed successfully, so I think it can be market as a duplicate. Thanks again!

[dnnnvx]

Duplicate of #3310 and fixed by #3597

[ashwaq06]

       title: '{{ template "slack.title" . }}'
          message: '{{ template "slack.text" . }}'

@dnnnvx I've attempted several methods to templatize the alerts while using the same kube-prometheus-stack Helm chart as yours. Specifically, I've tried creating templates through a ConfigMap and volume mounting it, but those attempts haven't worked. Additionally, I'm encountering same issue with configuring Discord webhooks for alert notifications. Can anyone provide guidance on how to properly set this up when using Alertmanager through the prometheus-stack Helm chart?
Any help or suggestions would be greatly appreciated