Alert metric reports different results to what the user sees via API

[gotjosh]

Fixes #1439 and #2619 (which are duplicates)

The previous metric is not technically reporting incorrect results as the alerts are still around and will be re-used if that same alert (equal fingerprint) is received before it is GCed. Therefore, I have kept the old metric under a new name alertmanager_marked_alerts and repurposed the current metric to match what the user sees in the UI.

I have tested this (and you can too) with the following:

1. First, send a payload to the Alertmanager with different alerts to cover all the cases (make sure you edit the endsAt for AlwaysFiringExpired, AlwaysFiringExpiringSoon and others as necessary) - you want one alert to be expired on reception and one that eventually expires to see how the metrics behave.

## Request
curl -X "POST" "http://localhost:9093/api/v2/alerts" \
     -H 'Content-Type: text/plain; charset=utf-8' \
     -d $'[
  {
    "labels": {
      "alertname": "AlwaysFiringExpiringSoon",
      "cluster": "us-central1",
      "namespace": "app2"
    },
    "generatorURL": "http://url.com",
    "endsAt": "2022-06-10 12:55:40"
  },
  {
    "labels": {
      "alertname": "AlwaysFiringExpired",
      "cluster": "us-central1",
      "namespace": "app2"
    },
    "generatorURL": "http://url.com",
    "endsAt": "2022-06-10 10:40:40"
  },
  {
    "labels": {
      "alertname": "AlwaysFiringToBeSilenced",
      "cluster": "us-central1",
      "namespace": "app2"
    },
    "generatorURL": "http://url.com",
    "endsAt": "2023-06-11 10:40:40"
  },
  {
    "labels": {
      "alertname": "AlwaysFiringActive",
      "cluster": "us-central1",
      "namespace": "app2"
    },
    "generatorURL": "http://url.com",
    "endsAt": "2023-06-11 10:40:40"
  }
]'

2. Make sure you set up silence for AlwaysFiringToBeSilenced.
3. In the UI, you should only see 1 alert after the silence and both expired alerts have disappeared: AlwaysFiringActive.
4. With this, the metrics are now accurate. Reporting 4 for the marker metrics but only 2 (one suppressed and 1 active for the new metric)

Marker

New alertmanager_alerts

[gotjosh]

Moved alerts because mtx is not protecting it and this was a bit confusing at first. alerts has its own mutex.

[gotjosh]

Kept the old metric intact functionality-wise, just added more label to complete the story.

[gotjosh]

Needs #2944

[gotjosh]

This is the big change, we now skip resolved alerts.

[roidelapluie]

This LGTM.

[gotjosh]

@roidelapluie @simonpasquier I have just rebased - this should be all good.

[roidelapluie]

$ curl -s 127.0.0.1:9093/api/v2/alerts|jq '. | length'
481
$ curl -s 127.0.0.1:9093/metrics|grep 'alertmanager\(_marked\)\?_alerts'
# HELP alertmanager_alerts How many alerts by state.
# TYPE alertmanager_alerts gauge
alertmanager_alerts{state="active"} 481
alertmanager_alerts{state="suppressed"} 0
alertmanager_alerts{state="unprocessed"} 0
# HELP alertmanager_alerts_invalid_total The total number of received alerts that were invalid.
# TYPE alertmanager_alerts_invalid_total counter
alertmanager_alerts_invalid_total{version="v1"} 0
alertmanager_alerts_invalid_total{version="v2"} 0
# HELP alertmanager_alerts_received_total The total number of received alerts.
# TYPE alertmanager_alerts_received_total counter
alertmanager_alerts_received_total{status="firing",version="v1"} 0
alertmanager_alerts_received_total{status="firing",version="v2"} 5475
alertmanager_alerts_received_total{status="resolved",version="v1"} 0
alertmanager_alerts_received_total{status="resolved",version="v2"} 18
# HELP alertmanager_marked_alerts How many alerts by state are currently marked in the Alertmanager regardless of their expiry.
# TYPE alertmanager_marked_alerts gauge
alertmanager_marked_alerts{state="active"} 489
alertmanager_marked_alerts{state="suppressed"} 0
alertmanager_marked_alerts{state="unprocessed"} 0

[roidelapluie]

Thanks!

[roidelapluie]

More results, at the 500 alerts mark

500


alertmanager_alerts{state="active"} 500
alertmanager_alerts{state="suppressed"} 0
alertmanager_alerts{state="unprocessed"} 0
alertmanager_marked_alerts{state="active"} 534
alertmanager_marked_alerts{state="suppressed"} 0
alertmanager_marked_alerts{state="unprocessed"} 0

LGTM, I will stop my test.