Add documentation for UTF-8 strict mode (#8596) (#8615)

This commit adds documentation for how to enable UTF-8 strict mode
in the Grafana Mimir Alertmanager, including both new and existing
Mimir installations.

(cherry picked from commit 827833a)

Co-authored-by: George Robinson <[email protected]>

docs/sources/mimir/references/architecture/components/alertmanager.md

@@ -156,3 +156,70 @@ You must configure the [ruler]({{< relref "./ruler" >}}) with the addresses of A
 Point the address to Alertmanager’s API.
 You can configure Alertmanager’s API prefix via the `-http.alertmanager-http-prefix` flag, which defaults to `/alertmanager`.
 For example, if Alertmanager is listening at `http://mimir-alertmanager.namespace.svc.cluster.local` and it is using the default API prefix, set `-ruler.alertmanager-url` to `http://mimir-alertmanager.namespace.svc.cluster.local/alertmanager`.
+
+## Enable UTF-8
+
+In effort to support alerts from [OpenTelemetry](https://opentelemetry.io/) (OTel) data, [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) has added support for UTF-8. This is supported as an opt-in feature for the Grafana Mimir Alertmanager in Mimir versions 2.12 and later.
+
+{{< admonition type="warning" >}}
+Enabling and then disabling UTF-8 strict mode can break existing tenant configurations if tenants added UTF-8 characters to their Alertmanager configuration while it was enabled. Once enabled, disable UTF-8 strict mode with caution.
+{{< /admonition >}}
+
+For new Mimir installations, enable support for UTF-8 before creating any tenant configurations. You can do this by changing [`utf8-strict-mode-enabled`]({{< relref "./../../../configure/configuration-parameters#alertmanager" >}}) to `true`.
+
+For existing Mimir installations, there are a number of breaking changes that might affect existing tenant configurations. Follow these instructions to ensure all existing tenant configurations are compatible with UTF-8 before enabling it.
+
+### What are the breaking changes?
+
+In order to support UTF-8, Alertmanager has added a new parser for label matchers (often abbreviated as matchers), which has a number of breaking changes.
+
+{{< admonition type="note" >}}
+If you are unfamiliar with what matchers are or how they are used in a tenant configuration, you can find more information about them in the [Prometheus Alertmanager documentation](https://prometheus.io/docs/alerting/latest/configuration).
+{{< /admonition >}}
+
+Grafana Mimir provides a number of tools to help you identify whether any existing tenant configurations are affected by these breaking changes, and to migrate any affected tenant configurations in a way that is backwards-compatible, doesn't change the behavior of existing matchers, and works even in Mimir installations that do not have UTF-8 enabled.
+
+### Identify affected tenant configurations
+
+To identify affected tenant configurations, take the following steps:
+
+1. Make sure Mimir is running version 2.12 or later with debug-level logging enabled.
+2. To identify any tenant configurations that are incompatible with UTF-8 (meaning the tenant configuration fails to load and the [fallback configuration](#fallback-configuration) is used instead), search Mimir server logs for lines containing `Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible`. Each log line includes the invalid matcher from the tenant configuration and the ID of the affected tenant. For example:
+
+   ```
+   msg="Alertmanager is moving to a new parser for labels and matchers, and this input is incompatible. Alertmanager has instead parsed the input using the classic matchers parser as a fallback. To make this input compatible with the UTF-8 matchers parser please make sure all regular expressions and values are double-quoted. If you are still seeing this message please open an issue." input="foo=" origin=api err="end of input: expected label value" suggestion="foo=\"\"" user="1"
+   ```
+
+   In this example, the tenant with User ID `1` has an incompatible matcher in their tenant configuration `foo=` and should to be changed to the suggestion `foo=""`.
+
+3. To identify any tenant configurations that are compatible with UTF-8 but contain matchers that might change in behavior when its enabled, search Mimir server logs for lines containing `Matchers input has disagreement`. Disagreement occurs when a matcher is valid, but due to adding support for UTF-8, it can behave differently when UTF-8 is enabled.
+
+   ```
+   msg="Matchers input has disagreement" input="foo=\"\\xf0\\x9f\\x99\\x82\"" user="1"
+   ```
+
+{{< admonition type="note" >}}
+It is possible for a tenant configuration to be both incompatible with UTF-8 and have disagreement, as an individual tenant configuration can contain a large number of matchers across different routes and inhibition rules.
+{{< /admonition >}}
+
+### Fix tenant configurations
+
+To fix any identified tenant configurations, take the following steps:
+
+1. Use the `migrate-utf8` [command]({{< relref "./../../../manage/tools/mimirtool#migrate-alertmanager-configuration-for-utf-8-in-mimir-212-and-later" >}}) in mimirtool to fix any tenant configurations that are incompatible with UTF-8. This command can migrate existing tenant configurations in a way that is backwards-compatible, doesn't change the behavior of existing matchers, and works even in Mimir installations that don't have UTF-8 enabled. If you cannot use mimirtool, you can edit tenant configurations by hand through applying each suggestion from the Mimir server logs.
+
+2. You must look at tenant configurations that have disagreement on a case-by-case basis. Depending on the nature of the disagreement, you might not need to fix a matcher with disagreement. For example `\xf0\x9f\x99\x82` is the byte sequence for the 🙂 emoji. If the intention is to match a literal 🙂 emoji then no change is required. However, if the intention is to match the literal `\xf0\x9f\x99\x82` then you need to change the matcher to use `\\xf0\\x9f\\x99\\x82` instead.
+
+{{< admonition type="note" >}}
+It's rare to find cases of disagreement in a tenant configuration, as most tenants do not need to match alerts that contain literal UTF-8 byte sequences in their labels.
+{{< /admonition >}}
+
+### Final steps
+
+1. After identifying and fixing all affected tenant configurations, check the Mimir server logs again to make sure you haven't missed any tenant configurations.
+
+2. You can now enable UTF-8 by setting [`utf8-strict-mode-enabled`]({{< relref "./../../../configure/configuration-parameters#alertmanager" >}}) to `true`.
+
+3. Any incompatible tenant configurations will fail to load. You can identify if any tenant configurations are failing to load by searching the Mimir server logs for lines containing `error applying config`, or querying the `alertmanager_config_last_reload_successful` gauage for `0`.
+
+4. You can disable debug-level logging.