feat(alertmanager): support loki alerts GeneratorURL in template functions

[fgouteroux]

Hello,

We used mimir (v2.10.0) and loki (2.9.1), and the loki ruler is configured to send alerts to mimir alertmanager.

In my mimir alertmanager logs, we got this error since the loki upgrade to 2.9.1:

Oct 05 09:28:39 mimiralert01.example.com mimir[2867549]: {"caller":"dispatch.go:352","component":"dispatcher","err":"test_preprod_receiver/pagerduty[0]: notify retry canceled due to unrecoverable error after 1 attempts: \"grafana_link\": failed to template \"{{ grafanaExploreURL \\\"https://my-grafana.example.com\\\" \\\"prometheus\\\" \\\"now-1h\\\" \\\"now\\\" (queryFromGeneratorURL (index .Alerts 0).GeneratorURL) }}\": template: :1:105: executing \"\" at <queryFromGeneratorURL (index .Alerts 0).GeneratorURL>: error calling queryFromGeneratorURL: query not found in the generator URL","insight":"true","level":"error","msg":"Notify for alerts failed","num_alerts":1,"ts":"2023-10-05T09:28:39.164215296Z","user":"test"}

Before the PR grafana/loki#8500 the loki GeneratorURL was starting with /graph?g0.expr and now it is /explore?left=, so now we got an error for loki alerts, caused by https://github.com/grafana/mimir/blob/main/pkg/alertmanager/alertmanager_template.go#L68

I would like to extend the mimir alertmanager templating function for grafanaExploreURL and queryFromGeneratorURL and support the generator url from loki. But I'm not sure that my implementation is the good way, as we can be confused with the alertmanager template function usage like:

grafana_link = "{{ grafanaExploreURL \"https://my-grafana.example.com\" \"prometheus\" \"now-1h\" \"now\" (queryFromGeneratorURL (index .Alerts 0).GeneratorURL) }}

As it mean that all grafana link will be use the prometheus datasource, except for alerts coming from loki.

Any helps appreciated.

@FUSAKLA as you did a good work with theses functions, maybe you can have a look ?

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[fgouteroux]

hello, any thoughts about this ?

[fgouteroux]

hello, no feedback on this ?

[stevesg]

Hi @fgouteroux - thanks for the PR. I'm taking a look now.

[stevesg]

If I understand right, what we want to accomplish is if the system (e.g. Loki) sending alerts already provide a Grafana Explore URL, then we want to pass that through unchanged.

I see the existing functions we have aren't really suitable for this purpose though. If we change queryFromGeneratorURL to pass-through the URL, then that's confusing given the name of the function. The expectation is that a query string is always returned, not a URL.

I'm not sure the existing functions can satisfy the need for passing through URLs, without breaking someone. So the options might well be:

1. Extend the existing functions following the existing pattern, but accept that we lose any "richness" from the URL (the big issue here I see being the datasource UID)
2. Adding a new function, that takes an arbitrary "GeneratorURL" and emits the "Explore URL". A descriptive but overly verbose name might be grafanaExploreURLFromGeneratorURL.
3. Fix it in the template itself (e.g. do the if prefixed ...)

I'll think on this a little more.

[fgouteroux]

hi, anything I can do to move forward on this PR ?

[fgouteroux]

Hi, any progress on the right direction about this ?

[fgouteroux]

hi @stevesg no chance to purpose a better way ?

[fgouteroux]

I found an other other way to support Loki alerts, I need an additionnal annotation or label and a conditionnal check in alertmanager config.

Using a template:

{{ define "grafana_link" -}}
    {{- if eq .CommonLabels.source_stack "Mimir" -}}
        {{ grafanaExploreURL "https://telemetry.example.com" "prometheus" (printf "%d000" ((index .Alerts 0).StartsAt.Add -900000000000).Unix) (printf "%d000" ((index .Alerts 0).StartsAt.Add 900000000000).Unix) (queryFromGeneratorURL (index .Alerts 0).GeneratorURL) }}
    {{- else if eq .CommonLabels.source_stack "Loki" -}}
        https://telemetry.example.com{{ html (index .Alerts 0).GeneratorURL | reReplaceAll "}" (printf ",\"range\":{\"from\":\"%d000\",\"to\":\"%d000\"}}" ((index .Alerts 0).StartsAt.Add -900000000000).Unix ((index .Alerts 0).StartsAt.Add 900000000000).Unix)}}
    {{- end -}}
{{- end -}}

then calling it:

grafana_link: '{{ template "grafana_link" . }}'

For Loki, I rewrite the url with adding the from/to range query parameters with alert startAt +/-15min.

[LukoJy3D]

@stevesg, maybe you could recheck this 🙏 Reusing Mimir alertmanager for Loki is also super convenient. However, it does not build the URL correctly (due to the different logql syntax?)

[LukoJy3D]

@fgouteroux, generatorUrl coming from Loki was not properly encoded, it was fixed by grafana/loki#15601 and should be released with 3.4.0, that should allow to use it?