From 12924e07e1d6be81a088ca5204d799bad615dfea Mon Sep 17 00:00:00 2001 From: Michael Sverdlik Date: Fri, 11 Mar 2022 12:21:24 +1100 Subject: [PATCH 1/2] Hint about potential high cardinality metrics Add warning about potential high cardinality metrics due to the file path being included in some of the metrics. Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus. This can result in increased storage, slow down of queries, extra costs and so on. Address what has been discussed in #5553. --- docs/sources/operations/observability.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/sources/operations/observability.md b/docs/sources/operations/observability.md index 1e6c4077cd94a..7f77dd2b2887c 100644 --- a/docs/sources/operations/observability.md +++ b/docs/sources/operations/observability.md @@ -82,6 +82,13 @@ exposed by Promtail at its `/metrics` endpoint. See Promtail's documentation on An example Grafana dashboard was built by the community and is available as dashboard [10004](https://grafana.com/dashboards/10004). +## Metrics cardinality + +Some of the metrics above are emitted per tracked file (active) with the file path included in the labels. +Depending on your setup, this may result in very large amount of label values across the environment (cardinality). This is generally discouraged by [Prometheus](https://prometheus.io/docs/practices/naming/#labels) and may result with unwanted side effects. +Make sure to review the emitted metrics before starting scraping with Prometheus and config the scraping accordingly if you suspect this might be an issue. + + ## Mixins The Loki repository has a [mixin](https://github.com/grafana/loki/blob/master/production/loki-mixin) that includes a From 87b474d513348f3b05d4719822ddca2d7add48f6 Mon Sep 17 00:00:00 2001 From: Michael Sverdlik Date: Wed, 30 Mar 2022 14:31:49 +1100 Subject: [PATCH 2/2] Reword observability.md Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com> --- docs/sources/operations/observability.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/sources/operations/observability.md b/docs/sources/operations/observability.md index 7f77dd2b2887c..433517829a7a3 100644 --- a/docs/sources/operations/observability.md +++ b/docs/sources/operations/observability.md @@ -84,9 +84,9 @@ dashboard [10004](https://grafana.com/dashboards/10004). ## Metrics cardinality -Some of the metrics above are emitted per tracked file (active) with the file path included in the labels. -Depending on your setup, this may result in very large amount of label values across the environment (cardinality). This is generally discouraged by [Prometheus](https://prometheus.io/docs/practices/naming/#labels) and may result with unwanted side effects. -Make sure to review the emitted metrics before starting scraping with Prometheus and config the scraping accordingly if you suspect this might be an issue. +Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels. +This increases the quantity of label values across the environment, thereby increasing cardinality. Best practices with Prometheus [labels](https://prometheus.io/docs/practices/naming/#labels) discourage increasing cardinality in this way. +Review your emitted metrics before scraping with Prometheus, and configure the scraping to avoid this issue. ## Mixins