Reintroduce thanos_objstore_bucket_operation_failures_total alert

Metric was removed on PR 2002 by mistake Signed-off-by: dmaiocchi <[email protected]>
thanos-io · Dec 10, 2020 · babac04 · babac04
1 parent 4abb51a
commit babac04
Show file tree

Hide file tree

Showing 3 changed files with 23 additions and 0 deletions.
diff --git a/examples/alerts/alerts.md b/examples/alerts/alerts.md
@@ -314,6 +314,14 @@ rules:
   for: 5m
   labels:
     severity: critical
+- alert: ThanosSidecarBucketOperationsFailed
+  expr: rate(thanos_objstore_bucket_operation_failures_total{name="prometheus"}[5m]) > 0
+  for: 5m
+  labels:
+    team: TEAM
+  annotations:
+    summary: Thanos Sidecar bucket operations are failing
+    runbook_url: https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarunbucketoperationfailed
 - alert: ThanosSidecarUnhealthy
   annotations:
     description: Thanos Sidecar {{$labels.job}} {{$labels.pod}} is unhealthy for {{

diff --git a/mixin/alerts/sidecar.libsonnet b/mixin/alerts/sidecar.libsonnet
@@ -22,6 +22,20 @@
               severity: 'critical',
             },
           },
+          {
+            alert: 'ThanosSidecarBucketOperationsFailed',
+            annotations: {
+              description: 'Thanos Sidecar {{$labels.job}} {{$labels.pod}} bucket operations are failing',
+              summary: 'Thanos Sidecar cannot connect to Prometheus',
+            },
+            expr: |||
+              rate(thanos_objstore_bucket_operation_failures_total{name="prometheus"}[5m]) > 0
+            ||| % thanos.sidecar,
+            'for': '5m',
+            labels: {
+              severity: 'critical',
+            },
+          },
           {
             alert: 'ThanosSidecarUnhealthy',
             annotations: {

diff --git a/mixin/runbook.md b/mixin/runbook.md
@@ -85,6 +85,7 @@
 |Name|Summary|Description|Severity|Runbook|
 |---|---|---|---|---|
 |ThanosSidecarPrometheusDown|Thanos Sidecar cannot connect to Prometheus|Thanos Sidecar {{$labels.job}} {{$labels.pod}} cannot connect to Prometheus.|critical|[https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarprometheusdown](https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarprometheusdown)|
+|ThanosSideCarBucketOperationsFailed|Thanos Sidecar bucket operations are failing {{l$labels.job}} {{$labels.pod}}|critical|[https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarbucketoperationfailed](https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarbucketoperationfailed)|
 |ThanosSidecarUnhealthy|Thanos Sidecar is unhealthy.|Thanos Sidecar {{$labels.job}} {{$labels.pod}} is unhealthy for {{ $value }} seconds.|critical|[https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarunhealthy](https://github.com/thanos-io/thanos/tree/master/mixin/runbook.md#alert-name-thanossidecarunhealthy)|
 
 ## thanos-store.rules