From fa1e9a78a88a1370fc318387d3f74eb7e91dd859 Mon Sep 17 00:00:00 2001 From: Weifeng Wang Date: Wed, 17 Apr 2024 21:22:45 +0800 Subject: [PATCH 1/2] [Module components] Add provides receivers module Signed-off-by: Weifeng Wang --- modules/provider/README.md | 121 +++++++++++++++++++++++++++ modules/provider/grafana_cloud.alloy | 111 ++++++++++++++++++++++++ modules/provider/self_hosted.alloy | 110 ++++++++++++++++++++++++ 3 files changed, 342 insertions(+) create mode 100644 modules/provider/README.md create mode 100644 modules/provider/grafana_cloud.alloy create mode 100644 modules/provider/self_hosted.alloy diff --git a/modules/provider/README.md b/modules/provider/README.md new file mode 100644 index 0000000..2144d64 --- /dev/null +++ b/modules/provider/README.md @@ -0,0 +1,121 @@ +# Module Components + +This module provides receivers components for collecting data(`logs` `metrics` `traces` `profiles`). + +## Components + - [`self_hosted_stack`](#self_hosted_stack) + - [`grafana_cloud`](#grafana_cloud) + +### `self_hosted_stack` + +Module component to configure receivers for Self Hosted LGTMP Stack. + +***Arguments*** + +| Name | Required | Default | Description | +|:------------------------|:---------|:------------------------------------|:------------------------------------| +| `metrics_endpoint_url` | _no_ | `http://mimir:8080/api/v1/push` | Where to send collected `metrics`. | +| `logs_endpoint_url` | _no_ | `http://loki:3100/loki/api/v1/push` | Where to send collected `logs`. | +| `traces_endpoint_url` | _no_ | `http://tempo:4318` | Where to send collected `traces`. | +| `profiles_endpoint_url` | _no_ | `http://pyroscope:4040` | Where to send collected `profiles`. | + +***Exports*** + +| Name | Type | Description | +|---------------------|--------------------------|------------------------------------------------------------------| +| `metrics_receiver` | `prometheus.Interceptor` | A value that other components can use to send metrics data to. | +| `logs_receiver` | `loki.LogsReceiver` | A value that other components can use to send logs data to. | +| `traces_receiver` | `otelcol.Consumer` | A value that other components can use to send trace data to. | +| `profiles_receiver` | `write.fanOutClient` | A value that other components can use to send profiling data to. | + +***Example*** + +```alloy +import.git "provider" { + repository = "https://github.com/grafana/alloy-modules" + revision = "main" + path = "modules/provider" + pull_frequency = "24h" +} + +// get the receivers from provider +provider.self_hosted_stack "compose" { + metrics_endpoint_url = "http://mimir:8080/api/v1/push" +} + +// get the receivers from provider +provider.self_hosted_stack "kubernetes" { + metrics_endpoint_url = "http://mimir.monitoring-system.svc.cluster.local:8080/api/v1/push" +} + +// scrape metrics and write to metric receiver +prometheus.scrape "default" { + targets = [ + {"__address__" = "127.0.0.1:12345"}, + ] + + forward_to = [ + provider.self_hosted_stack.compose.metrics_receiver, + provider.self_hosted_stack.kubernetes.metrics_receiver, + ] +} +``` + +### `grafana_cloud` + +Module component to automatically configure receivers for Grafana Cloud. + +To create a token: + +1. Navigate to the [Grafana Cloud Portal](https://grafana.com/profile/org) +2. Go to either the `Access Policies` or `API Keys` page, located in the `Security` section +3. Create an Access Policy or API token with the correct permissions + +The token must have permissions to read stack information. The setup of these permissions depends on the type of token: + +- Access Policies need the `stacks:read` scope +- API Keys need at least the the `MetricsPublisher` role + +***Arguments*** + +| Name | Required | Default | Description | +|:-------------|:---------|:--------|:---------------------------------------------------| +| `stack_name` | _yes_ | `N/A` | Name of your stack as shown in the account console | +| `token` | _yes_ | `N/A` | Access policy token or API Key. | + +***Exports*** + +| Name | Type | Description | +|---------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------| +| `metrics_receiver` | `prometheus.Interceptor` | A value that other components can use to send metrics data to. | +| `logs_receiver` | `loki.LogsReceiver` | A value that other components can use to send logs data to. | +| `traces_receiver` | `otelcol.Consumer` | A value that other components can use to send trace data to. | +| `profiles_receiver` | `write.fanOutClient` | A value that other components can use to send profiling data to. | +| `stack_information` | `object` | Decoded representation of the [Stack info endpoint](https://grafana.com/docs/grafana-cloud/api-reference/cloud-api/#stacks). | + +***Example*** + +```alloy +import.git "provider" { + repository = "https://github.com/grafana/alloy-modules" + revision = "main" + path = "modules/provider" + pull_frequency = "24h" +} + +// get the receivers from provider +provider.grafana_cloud "stack_name" { + stack_name = env("GRAFANA_CLOUD_STACK_NAME") + token = env("GRAFANA_CLOUD_TOKEN") +} + +// scrape metrics and write to metric receiver +prometheus.scrape "default" { + targets = [ + {"__address__" = "127.0.0.1:12345"}, + ] + forward_to = [ + provider.grafana_cloud.stack_name.metrics_receiver, + ] +} +``` diff --git a/modules/provider/grafana_cloud.alloy b/modules/provider/grafana_cloud.alloy new file mode 100644 index 0000000..86a0b27 --- /dev/null +++ b/modules/provider/grafana_cloud.alloy @@ -0,0 +1,111 @@ +/* +Module Components: grafana_cloud +Description: Grafana Cloud Receiver Provider +*/ + +declare "grafana_cloud" { + /* + To create a token: + 1. Navigate to the Grafana Cloud Portal: https://grafana.com/profile/org + 2. Go to either the Access Policies or API Keys page, located in the Security section + 3. Create an Access Policy or API token with the correct permissions + + The token must have permissions to read stack information. The setup of these permissions depends on the type of token: + Access Policies need the stacks:read scope + API Keys need at least the the MetricsPublisher role + */ + + /*************************************************** + * ARGUMENTS + ***************************************************/ + argument "stack_name" { + comment = "Name of your stack as shown in the account console" + } + + argument "token" { + comment = "Access policy token or API Key." + } + + /*************************************************** + * External information + ***************************************************/ + remote.http "json_config" { + url = "https://grafana.com/api/instances/" + argument.stack_name.value + + client { + bearer_token = argument.token.value + } + poll_frequency = "24h" + } + + /*************************************************** + * Setup Receivers + ***************************************************/ + prometheus.remote_write "grafana_cloud" { + endpoint { + url = json_decode(remote.http.json_config.content)["hmInstancePromUrl"] + "/api/prom/push" + + basic_auth { + username = json_decode(remote.http.json_config.content)["hmInstancePromId"] + password = argument.token.value + } + } + } + + loki.write "grafana_cloud" { + endpoint { + url = json_decode(remote.http.json_config.content)["hlInstanceUrl"] + "/loki/api/v1/push" + + basic_auth { + username = json_decode(remote.http.json_config.content)["hlInstanceId"] + password = argument.token.value + } + } + } + + otelcol.auth.basic "grafana_cloud" { + username = json_decode(remote.http.json_config.content)["htInstanceId"] + password = argument.token.value + } + + otelcol.exporter.otlp "grafana_cloud" { + client { + endpoint = json_decode(remote.http.json_config.content)["htInstanceUrl"] + ":443" + auth = otelcol.auth.basic.grafana_cloud.handler + } + } + + pyroscope.write "grafana_cloud" { + endpoint { + url = json_decode(remote.http.json_config.content)["hpInstanceUrl"] + + basic_auth { + username = json_decode(remote.http.json_config.content)["hpInstanceId"] + password = argument.token.value + } + } + } + + /*************************************************** + * EXPORTS + ***************************************************/ + export "metrics_receiver" { + value = prometheus.remote_write.grafana_cloud.receiver + } + + export "logs_receiver" { + value = loki.write.grafana_cloud.receiver + } + + export "traces_receiver" { + value = otelcol.exporter.otlp.grafana_cloud.input + } + + export "profiles_receiver" { + value = pyroscope.write.grafana_cloud.receiver + } + + export "stack_information" { + value = json_decode(remote.http.json_config.content) + } +} diff --git a/modules/provider/self_hosted.alloy b/modules/provider/self_hosted.alloy new file mode 100644 index 0000000..286b17c --- /dev/null +++ b/modules/provider/self_hosted.alloy @@ -0,0 +1,110 @@ +/* +Module Components: self_hosted_stack +Description: Self Hosted Receiver Provider +*/ + +declare "self_hosted_stack" { + + /*************************************************** + * ARGUMENTS + ***************************************************/ + argument "metrics_endpoint_url" { + comment = "Where to send collected metrics." + optional = true + default = coalesce(env("SELF_HOSTED_METRICS_ENDPOINT_URL"), "http://mimir:8080/api/v1/push") + } + + argument "logs_endpoint_url" { + comment = "Where to send collected logs." + optional = true + default = coalesce(env("SELF_HOSTED_LOGS_ENDPOINT_URL"), "http://loki:3100/loki/api/v1/push") + } + + argument "traces_endpoint_url" { + comment = "Where to send collected traces." + optional = true + default = coalesce(env("SELF_HOSTED_TRACES_ENDPOINT_URL"), "http://tempo:4318") + } + + argument "profiles_endpoint_url" { + comment = "Where to send collected profiles." + optional = true + default = coalesce(env("SELF_HOSTED_PROFILES_ENDPOINT_URL"), "http://pyroscope:4040") + } + + /*************************************************** + * Setup Receivers + ***************************************************/ + prometheus.remote_write "self_hosted" { + endpoint { + url = argument.metrics_endpoint_url.value + send_native_histograms = true + + basic_auth { + password_file = coalesce(env("SELF_HOSTED_METRICS_BASIC_AUTH_PASSWORD_FILE"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD_FILE")) + password = coalesce(env("SELF_HOSTED_METRICS_BASIC_AUTH_PASSWORD"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD")) + username = coalesce(env("SELF_HOSTED_METRICS_BASIC_AUTH_USERNAME"), env("SELF_HOSTED_BASIC_AUTH_USERNAME")) + } + } + } + + loki.write "self_hosted" { + endpoint { + url = argument.logs_endpoint_url.value + + basic_auth { + password_file = coalesce(env("SELF_HOSTED_LOGS_BASIC_AUTH_PASSWORD_FILE"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD_FILE")) + password = coalesce(env("SELF_HOSTED_LOGS_BASIC_AUTH_PASSWORD"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD")) + username = coalesce(env("SELF_HOSTED_LOGS_BASIC_AUTH_USERNAME"), env("SELF_HOSTED_BASIC_AUTH_USERNAME")) + } + } + } + + otelcol.auth.basic "self_hosted" { + username = coalesce(env("SELF_HOSTED_TRACES_BASIC_AUTH_USERNAME"), env("SELF_HOSTED_BASIC_AUTH_USERNAME")) + password = coalesce(env("SELF_HOSTED_TRACES_BASIC_AUTH_PASSWORD"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD")) + } + + otelcol.exporter.otlphttp "self_hosted" { + client { + endpoint = argument.traces_endpoint_url.value + auth = otelcol.auth.basic.self_hosted.handler + + tls { + insecure = true + insecure_skip_verify = true + } + } + } + + pyroscope.write "self_hosted" { + endpoint { + url = argument.profiles_endpoint_url.value + + basic_auth { + password_file = coalesce(env("SELF_HOSTED_PROFILES_BASIC_AUTH_PASSWORD_FILE"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD_FILE")) + password = coalesce(env("SELF_HOSTED_PROFILES_BASIC_AUTH_PASSWORD"), env("SELF_HOSTED_BASIC_AUTH_PASSWORD")) + username = coalesce(env("SELF_HOSTED_PROFILES_BASIC_AUTH_USERNAME"), env("SELF_HOSTED_BASIC_AUTH_USERNAME")) + } + } + } + + /*************************************************** + * EXPORTS + ***************************************************/ + export "metrics_receiver" { + value = prometheus.remote_write.self_hosted.receiver + } + + export "logs_receiver" { + value = loki.write.self_hosted.receiver + } + + export "traces_receiver" { + value = otelcol.exporter.otlphttp.self_hosted.input + } + + export "profiles_receiver" { + value = pyroscope.write.self_hosted.receiver + } +} From bd8e68317240c87688edc48a7b871e9aacdb27df Mon Sep 17 00:00:00 2001 From: Weifeng Wang Date: Sat, 4 May 2024 13:00:21 +0800 Subject: [PATCH 2/2] fix markdownlint checks --- modules/provider/README.md | 27 ++++++++++++++------------- 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/modules/provider/README.md b/modules/provider/README.md index 2144d64..b4b9294 100644 --- a/modules/provider/README.md +++ b/modules/provider/README.md @@ -3,8 +3,9 @@ This module provides receivers components for collecting data(`logs` `metrics` `traces` `profiles`). ## Components - - [`self_hosted_stack`](#self_hosted_stack) - - [`grafana_cloud`](#grafana_cloud) + +- [`self_hosted_stack`](#self_hosted_stack) +- [`grafana_cloud`](#grafana_cloud) ### `self_hosted_stack` @@ -14,10 +15,10 @@ Module component to configure receivers for Self Hosted LGTMP Stack. | Name | Required | Default | Description | |:------------------------|:---------|:------------------------------------|:------------------------------------| -| `metrics_endpoint_url` | _no_ | `http://mimir:8080/api/v1/push` | Where to send collected `metrics`. | -| `logs_endpoint_url` | _no_ | `http://loki:3100/loki/api/v1/push` | Where to send collected `logs`. | -| `traces_endpoint_url` | _no_ | `http://tempo:4318` | Where to send collected `traces`. | -| `profiles_endpoint_url` | _no_ | `http://pyroscope:4040` | Where to send collected `profiles`. | +| `metrics_endpoint_url` | *no* | `http://mimir:8080/api/v1/push` | Where to send collected `metrics`. | +| `logs_endpoint_url` | *no* | `http://loki:3100/loki/api/v1/push` | Where to send collected `logs`. | +| `traces_endpoint_url` | *no* | `http://tempo:4318` | Where to send collected `traces`. | +| `profiles_endpoint_url` | *no* | `http://pyroscope:4040` | Where to send collected `profiles`. | ***Exports*** @@ -67,21 +68,21 @@ Module component to automatically configure receivers for Grafana Cloud. To create a token: -1. Navigate to the [Grafana Cloud Portal](https://grafana.com/profile/org) -2. Go to either the `Access Policies` or `API Keys` page, located in the `Security` section -3. Create an Access Policy or API token with the correct permissions +1. Navigate to the [Grafana Cloud Portal](https://grafana.com/profile/org) +2. Go to either the `Access Policies` or `API Keys` page, located in the `Security` section +3. Create an Access Policy or API token with the correct permissions The token must have permissions to read stack information. The setup of these permissions depends on the type of token: -- Access Policies need the `stacks:read` scope -- API Keys need at least the the `MetricsPublisher` role +- Access Policies need the `stacks:read` scope +- API Keys need at least the the `MetricsPublisher` role ***Arguments*** | Name | Required | Default | Description | |:-------------|:---------|:--------|:---------------------------------------------------| -| `stack_name` | _yes_ | `N/A` | Name of your stack as shown in the account console | -| `token` | _yes_ | `N/A` | Access policy token or API Key. | +| `stack_name` | *yes* | `N/A` | Name of your stack as shown in the account console | +| `token` | *yes* | `N/A` | Access policy token or API Key. | ***Exports***