From 6db13050aa78e365ec614126ef414734033c3408 Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Mon, 17 Jun 2019 15:58:45 +1000 Subject: [PATCH 1/7] [docs] Add initial aggregator documentation --- docs/how_to/aggregator.md | 159 ++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 160 insertions(+) create mode 100644 docs/how_to/aggregator.md diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md new file mode 100644 index 0000000000..048f210760 --- /dev/null +++ b/docs/how_to/aggregator.md @@ -0,0 +1,159 @@ +# Setting up m3aggregator + +## Introduction + +m3aggregator is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The m3coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round round fashion to multiple m3coordinators for the same metrics ingestion source (e.g. Prometheus server). + +Metrics sent to m3aggregator are correctly routed to the instance resposible for aggregating each metric and there is multiple m3aggregator replicas to ensure no single point of failure during aggregation. + +## Configuration + +Before setting up m3aggregator, make sure that you have at least [one M3DB node running](single_node.md) and a dedicated m3coordinator setup. + +We highly recommend running with at least a replication factor 2 for a m3aggregator deployment. + +### Running + +You can run m3aggregator by either building and running the binary yourself: + +``` +make m3aggregator +./bin/m3aggregator -f ./src/aggregator/config/m3aggregator.yml +``` + +Or you can run it with Docker using the Docker file located at `$GOPATH/src/github.com/m3db/m3/docker/m3aggregator/Dockerfile`. + +### Coordinator configuration + +Metrics will still arrive at the m3coordinator, they simply need to be forwarded to m3aggregators. The m3coordinator then also needs to receive metrics that have been aggregated from the m3aggregator and store them in M3DB, so running an ingestion server should be configured. + +Here is the config you should add to your m3coordinator: +``` +# This is for sending metrics to the remote m3aggregators +downsample: + remoteAggregator: + client: + placementKV: + namespace: /placement + placementWatcher: + key: m3aggregator + initWatchTimeout: 10s + hashType: murmur32 + shardCutoffLingerDuration: 1m + flushSize: 1440 + maxTimerBatchSize: 1120 + queueSize: 10000 + queueDropType: oldest + encoder: + initBufferSize: 2048 + maxMessageSize: 10485760 + bytesPool: + buckets: + - capacity: 2048 + count: 4096 + - capacity: 4096 + count: 4096 + watermark: + low: 0.7 + high: 1.0 + connection: + writeTimeout: 250ms + +# This is for configuring the ingestion server that will receive metrics from the m3aggregators on port 7507 +ingest: + ingester: + workerPoolSize: 100 + opPool: + size: 100 + retry: + maxRetries: 3 + jitter: true + logSampleRate: 0.01 + m3msg: + server: + listenAddress: "0.0.0.0:7507" + retry: + maxBackoff: 10s + jitter: true +``` + +### Topology + +Once you have your m3aggregators running, you can setup a m3aggregator topology by issuing a request to your coordinator (be sure to use your own hostnames, number of shards and replication factor): +``` +curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/services/m3aggregator/placement/init -d '{ + "num_shards": 64, + "replication_factor": 2, + "instances": [ + { + "id": "m3aggregator01:6000", + "isolation_group": "availability-zone-a", + "zone": "embedded", + "weight": 100, + "endpoint": "m3aggregator01:6000", + "hostname": "m3aggregator01", + "port": 6000 + }, + { + "id": "m3aggregator02:6000", + "isolation_group": "availability-zone-b", + "zone": "embedded", + "weight": 100, + "endpoint": "m3aggregator02:6000", + "hostname": "m3aggregator02", + "port": 6000 + } + ] +}' +``` + +### Aggregation topic and m3coordinator discovery + +#### Create aggregation topic + +Now we must setup a topic for the m3aggregator to send computed metrics aggregations back to an m3coordinator instance for storage to M3DB: +``` +curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/topic/init -d '{ + "numberOfShards": 64 +}' +``` + +#### Create m3coordinator consumer group for topic + +Then the m3coordinators need to be configured to receive traffic for this topic (note ingest at port 7507 must match the configured port for your m3coordinator ingest server): +``` +curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/services/m3coordinator/placement/init -d '{ + "instances": [ + { + "id": "m3coordinator01", + "zone": "embedded", + "endpoint": "m3coordinator01:7507", + "hostname": "m3coordinator01", + "port": 7507 + } + ] +}' +``` + +**Note:** When you add and remove m3coordinators they must be added to this topic. + +#### Add m3coordinator consumer group to topic + +Add the m3coordinator placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory message buffer): +``` +curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/topic -d '{ + "consumerService": { + "serviceId": { + "name": "m3coordinator", + "environment": "default_env", + "zone": "embedded" + }, + "consumptionType": "SHARED", + "messageTtlNanos": "300000000000" + } +}' +``` + +## Usage + +Send metrics as usual to your m3coordinators in round robin fashion (or any other load balancing strategy), the metrics will be forwarded to the m3aggregators, then once aggregated they will be sent to the m3coordinators to write to M3DB. diff --git a/mkdocs.yml b/mkdocs.yml index 491350c331..d6d2b699f8 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -86,6 +86,7 @@ pages: - "M3DB Cluster Deployment, Manually": "how_to/cluster_hard_way.md" - "M3DB on Kubernetes": "how_to/kubernetes.md" - "M3Query": "how_to/query.md" + - "M3Aggregator": "how_to/aggregator.md" - "Operational Guides": - "Replication and Deployment in Zones": "operational_guide/replication_and_deployment_in_zones.md" - "Tuning Availability, Consistency, and Durability": "operational_guide/availability_consistency_durability.md" From fdd2472ca3a14951a332b8a2bccbe0605c09d187 Mon Sep 17 00:00:00 2001 From: Evan Yin Date: Sun, 20 Oct 2019 23:25:30 -0700 Subject: [PATCH 2/7] Update aggregator setup guide --- docs/how_to/aggregator.md | 151 ++++++++++++++++++-------------------- 1 file changed, 73 insertions(+), 78 deletions(-) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index 048f210760..c53d72dd1b 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -12,19 +12,81 @@ Before setting up m3aggregator, make sure that you have at least [one M3DB node We highly recommend running with at least a replication factor 2 for a m3aggregator deployment. -### Running +### Topology -You can run m3aggregator by either building and running the binary yourself: +#### Initializing aggregator topology +You can setup a m3aggregator topology by issuing a request to your coordinator (be sure to use your own hostnames, number of shards and replication factor): +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/services/m3aggregator/placement/init -d '{ + "num_shards": 64, + "replication_factor": 2, + "instances": [ + { + "id": "m3aggregator01:6000", + "isolation_group": "availability-zone-a", + "zone": "embedded", + "weight": 100, + "endpoint": "m3aggregator01:6000", + "hostname": "m3aggregator01", + "port": 6000 + }, + { + "id": "m3aggregator02:6000", + "isolation_group": "availability-zone-b", + "zone": "embedded", + "weight": 100, + "endpoint": "m3aggregator02:6000", + "hostname": "m3aggregator02", + "port": 6000 + } + ] +}' +``` +#### Initializing m3msg topic for m3coordinator ingestion from m3aggregators +Now we must setup a topic for the m3aggregator to send computed metrics aggregations back to an m3coordinator instance for storage to M3DB: ``` -make m3aggregator -./bin/m3aggregator -f ./src/aggregator/config/m3aggregator.yml +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/topic/init -d '{ + "numberOfShards": 64 +}' ``` -Or you can run it with Docker using the Docker file located at `$GOPATH/src/github.com/m3db/m3/docker/m3aggregator/Dockerfile`. +#### Initializing m3coordinator topology +Then the m3coordinators need to be configured to receive traffic for this topic (note ingest at port 7507 must match the configured port for your m3coordinator ingest server): +``` +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/services/m3coordinator/placement/init -d '{ + "instances": [ + { + "id": "m3coordinator01", + "zone": "embedded", + "endpoint": "m3coordinator01:7507", + "hostname": "m3coordinator01", + "port": 7507 + } + ] +}' -### Coordinator configuration +``` +**Note:** When you add and remove m3coordinators they must be added to this placement. + +#### Add m3coordinator consumer group to topic +Add the m3coordinator placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory message buffer): +``` +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/topic -d '{ + "consumerService": { + "serviceId": { + "name": "m3coordinator", + "environment": "default_env", + "zone": "embedded" + }, + "consumptionType": "SHARED", + "messageTtlNanos": "300000000000" + } +}' +``` +### Running +#### Dedicated Coordinator Metrics will still arrive at the m3coordinator, they simply need to be forwarded to m3aggregators. The m3coordinator then also needs to receive metrics that have been aggregated from the m3aggregator and store them in M3DB, so running an ingestion server should be configured. Here is the config you should add to your m3coordinator: @@ -77,82 +139,15 @@ ingest: jitter: true ``` -### Topology - -Once you have your m3aggregators running, you can setup a m3aggregator topology by issuing a request to your coordinator (be sure to use your own hostnames, number of shards and replication factor): -``` -curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/services/m3aggregator/placement/init -d '{ - "num_shards": 64, - "replication_factor": 2, - "instances": [ - { - "id": "m3aggregator01:6000", - "isolation_group": "availability-zone-a", - "zone": "embedded", - "weight": 100, - "endpoint": "m3aggregator01:6000", - "hostname": "m3aggregator01", - "port": 6000 - }, - { - "id": "m3aggregator02:6000", - "isolation_group": "availability-zone-b", - "zone": "embedded", - "weight": 100, - "endpoint": "m3aggregator02:6000", - "hostname": "m3aggregator02", - "port": 6000 - } - ] -}' -``` - -### Aggregation topic and m3coordinator discovery - -#### Create aggregation topic - -Now we must setup a topic for the m3aggregator to send computed metrics aggregations back to an m3coordinator instance for storage to M3DB: -``` -curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/topic/init -d '{ - "numberOfShards": 64 -}' -``` - -#### Create m3coordinator consumer group for topic +#### Aggregator +You can run m3aggregator by either building and running the binary yourself: -Then the m3coordinators need to be configured to receive traffic for this topic (note ingest at port 7507 must match the configured port for your m3coordinator ingest server): ``` -curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/services/m3coordinator/placement/init -d '{ - "instances": [ - { - "id": "m3coordinator01", - "zone": "embedded", - "endpoint": "m3coordinator01:7507", - "hostname": "m3coordinator01", - "port": 7507 - } - ] -}' +make m3aggregator +./bin/m3aggregator -f ./src/aggregator/config/m3aggregator.yml ``` -**Note:** When you add and remove m3coordinators they must be added to this topic. - -#### Add m3coordinator consumer group to topic - -Add the m3coordinator placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory message buffer): -``` -curl -vvvsSf -X POST http://m3coordinator01:7201/api/v1/topic -d '{ - "consumerService": { - "serviceId": { - "name": "m3coordinator", - "environment": "default_env", - "zone": "embedded" - }, - "consumptionType": "SHARED", - "messageTtlNanos": "300000000000" - } -}' -``` +Or you can run it with Docker using the Docker file located at `$GOPATH/src/github.com/m3db/m3/docker/m3aggregator/Dockerfile`. ## Usage From eca080843c476e3ce8f654dbb11c25755e6419b2 Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Tue, 4 Aug 2020 20:13:08 -0400 Subject: [PATCH 3/7] Update documentation --- docs/how_to/aggregator.md | 148 ++++++++++++++++++++++++-------------- 1 file changed, 95 insertions(+), 53 deletions(-) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index c53d72dd1b..e5f4413dfc 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -1,23 +1,24 @@ -# Setting up m3aggregator +# Setting up M3 Aggregator ## Introduction -m3aggregator is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The m3coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round round fashion to multiple m3coordinators for the same metrics ingestion source (e.g. Prometheus server). +`m3aggregator` is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The M3 Coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round round fashion to multiple M3 Coordinators for the same metrics ingestion source (e.g. Prometheus server). -Metrics sent to m3aggregator are correctly routed to the instance resposible for aggregating each metric and there is multiple m3aggregator replicas to ensure no single point of failure during aggregation. +Metrics sent to `m3aggregator` are correctly routed to the instance resposible for aggregating each metric and there is multiple m3aggregator replicas to ensure no single point of failure during aggregation. ## Configuration Before setting up m3aggregator, make sure that you have at least [one M3DB node running](single_node.md) and a dedicated m3coordinator setup. -We highly recommend running with at least a replication factor 2 for a m3aggregator deployment. +We highly recommend running with at least a replication factor 2 for a `m3aggregator` deployment. ### Topology #### Initializing aggregator topology + You can setup a m3aggregator topology by issuing a request to your coordinator (be sure to use your own hostnames, number of shards and replication factor): ```bash -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/services/m3aggregator/placement/init -d '{ +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/services/m3aggregator/placement/init -d '{ "num_shards": 64, "replication_factor": 2, "instances": [ @@ -43,18 +44,58 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST }' ``` -#### Initializing m3msg topic for m3coordinator ingestion from m3aggregators -Now we must setup a topic for the m3aggregator to send computed metrics aggregations back to an m3coordinator instance for storage to M3DB: +#### Initializing m3msg topic for m3aggregator to receive from m3coordinators to aggregate metrics + +Now we must setup a topic for the `m3aggregator` to receive unaggregated metrics from `m3coordinator` instances: + +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregator_ingest" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic/init -d '{ + "numberOfShards": 64 +}' ``` -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/topic/init -d '{ + +#### Initializing m3msg topic for m3coordinator to receive from m3aggregators to write to M3DB + +Now we must setup a topic for the `m3aggregator` to send computed metrics aggregations back to an `m3coordinator` instance for storage to M3DB: +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic/init -d '{ "numberOfShards": 64 }' ``` -#### Initializing m3coordinator topology -Then the m3coordinators need to be configured to receive traffic for this topic (note ingest at port 7507 must match the configured port for your m3coordinator ingest server): +#### Add m3aggregagtor consumer group to ingest topic + +Add the `m3aggregator` placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory retry message buffer): +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregator_ingest" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ + "consumerService": { + "serviceId": { + "name": "m3aggregator", + "environment": "namespace/m3db-cluster-name", + "zone": "embedded" + }, + "consumptionType": "REPLICATED", + "messageTtlNanos": "300000000000" + } +}' +``` + +**Note:** 300000000000 nanoseconds is a TTL of 5 minutes for messages to rebuffer for retry. + +#### Initializing m3msg topic for m3coordinator to receive from m3aggregator to write to M3DB + +Now we must setup a topic for the `m3coordinator` to receive unaggregated metrics from `m3aggregator` instances to write to M3DB: +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregated_metrics" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic/init -d '{ + "numberOfShards": 64 +}' ``` -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/services/m3coordinator/placement/init -d '{ + +#### Initializing m3coordinator topology + +Then `m3coordinator` instances need to be configured to receive traffic for this topic (note ingest at port 7507 must match the configured port for your `m3coordinator` ingest server, see config at bottom of this guide): +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/services/m3coordinator/placement/init -d '{ "instances": [ { "id": "m3coordinator01", @@ -65,18 +106,19 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST } ] }' - ``` -**Note:** When you add and remove m3coordinators they must be added to this placement. -#### Add m3coordinator consumer group to topic -Add the m3coordinator placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory message buffer): -``` -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3coordinator01:7201/api/v1/topic -d '{ +**Note:** When you add or remove `m3coordinator` instances they must be added to this placement. + +#### Add m3coordinator consumer group to outbound topic + +Add the `m3coordinator` placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory retry message buffer): +```bash +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ "consumerService": { "serviceId": { "name": "m3coordinator", - "environment": "default_env", + "environment": "namespace/m3db-cluster-name", "zone": "embedded" }, "consumptionType": "SHARED", @@ -85,48 +127,47 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST }' ``` +**Note:** 300000000000 nanoseconds is a TTL of 5 minutes for messages to rebuffer for retry. + ### Running + #### Dedicated Coordinator -Metrics will still arrive at the m3coordinator, they simply need to be forwarded to m3aggregators. The m3coordinator then also needs to receive metrics that have been aggregated from the m3aggregator and store them in M3DB, so running an ingestion server should be configured. -Here is the config you should add to your m3coordinator: -``` +Metrics will still arrive at the `m3coordinator`, they simply need to be forwarded to an `m3aggregator`. The `m3coordinator` then also needs to receive metrics that have been aggregated from the `m3aggregator` and store them in M3DB, so running an ingestion server should be configured. + +Here is the config you should add to your `m3coordinator`: +```yaml # This is for sending metrics to the remote m3aggregators downsample: remoteAggregator: client: - placementKV: - namespace: /placement - placementWatcher: - key: m3aggregator - initWatchTimeout: 10s - hashType: murmur32 - shardCutoffLingerDuration: 1m - flushSize: 1440 - maxTimerBatchSize: 1120 - queueSize: 10000 - queueDropType: oldest - encoder: - initBufferSize: 2048 - maxMessageSize: 10485760 - bytesPool: - buckets: - - capacity: 2048 - count: 4096 - - capacity: 4096 - count: 4096 - watermark: - low: 0.7 - high: 1.0 - connection: - writeTimeout: 250ms + type: m3msg + m3msg: + producer: + writer: + topicName: aggregator_ingest + topicServiceOverride: + zone: embedded + environment: namespace/m3db-cluster-name + placement: + isStaged: true + placementServiceOverride: + namespaces: + placement: /placement + connection: + numConnections: 4 + messagePool: + size: 16384 + watermark: + low: 0.2 + high: 0.5 # This is for configuring the ingestion server that will receive metrics from the m3aggregators on port 7507 ingest: ingester: - workerPoolSize: 100 + workerPoolSize: 10000 opPool: - size: 100 + size: 10000 retry: maxRetries: 3 jitter: true @@ -139,16 +180,17 @@ ingest: jitter: true ``` -#### Aggregator -You can run m3aggregator by either building and running the binary yourself: +#### M3 Aggregator -``` +You can run `m3aggregator` by either building and running the binary yourself: + +```bash make m3aggregator ./bin/m3aggregator -f ./src/aggregator/config/m3aggregator.yml ``` -Or you can run it with Docker using the Docker file located at `$GOPATH/src/github.com/m3db/m3/docker/m3aggregator/Dockerfile`. +Or you can run it with Docker using the Docker file located at `docker/m3aggregator/Dockerfile` or the publicly provided image `quay.io/m3db/m3aggregator:latest`. ## Usage -Send metrics as usual to your m3coordinators in round robin fashion (or any other load balancing strategy), the metrics will be forwarded to the m3aggregators, then once aggregated they will be sent to the m3coordinators to write to M3DB. +Send metrics as usual to your `m3coordinator` instances in round robin fashion (or any other load balancing strategy), the metrics will be forwarded to the `m3aggregator` instances, then once aggregated they will be returned to the `m3coordinator` instances to write to M3DB. From b99aa65036be20100ca2d3595a569ae6ada10c7e Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Tue, 4 Aug 2020 20:18:15 -0400 Subject: [PATCH 4/7] Add config example --- docs/how_to/aggregator.md | 221 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 221 insertions(+) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index e5f4413dfc..2ac3c0b046 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -191,6 +191,227 @@ make m3aggregator Or you can run it with Docker using the Docker file located at `docker/m3aggregator/Dockerfile` or the publicly provided image `quay.io/m3db/m3aggregator:latest`. +You can use a config like so, making note of the topics used such as `aggregator_ingest` and `aggregated_metrics` and the corresponding environment `namespace/m3db-cluster-name`: + +```yaml +logging: + level: info + +metrics: + scope: + prefix: m3aggregator + prometheus: + onError: none + handlerPath: /metrics + listenAddress: 0.0.0.0:6002 + timerType: histogram + sanitization: prometheus + samplingRate: 1.0 + extended: none + +m3msg: + server: + listenAddress: 0.0.0.0:6000 + retry: + maxBackoff: 10s + jitter: true + consumer: + messagePool: + size: 16384 + watermark: + low: 0.2 + high: 0.5 + +http: + listenAddress: 0.0.0.0:6001 + readTimeout: 60s + writeTimeout: 60s + +kvClient: + etcd: + env: namespace/m3db-cluster-name + zone: embedded + service: m3aggregator + cacheDir: /var/lib/m3kv + etcdClusters: + - zone: embedded + endpoints: + - dbnode01:2379 + +runtimeOptions: + kvConfig: + environment: namespace/m3db-cluster-name + zone: embedded + writeValuesPerMetricLimitPerSecondKey: write-values-per-metric-limit-per-second + writeValuesPerMetricLimitPerSecond: 0 + writeNewMetricLimitClusterPerSecondKey: write-new-metric-limit-cluster-per-second + writeNewMetricLimitClusterPerSecond: 0 + writeNewMetricNoLimitWarmupDuration: 0 + +aggregator: + hostID: + resolver: environment + envVarName: M3AGGREGATOR_HOST_ID + instanceID: + type: host_id + verboseErrors: true + metricPrefix: "" + counterPrefix: "" + timerPrefix: "" + gaugePrefix: "" + aggregationTypes: + counterTransformFnType: empty + timerTransformFnType: suffix + gaugeTransformFnType: empty + aggregationTypesPool: + size: 1024 + quantilesPool: + buckets: + - count: 256 + capacity: 4 + - count: 128 + capacity: 8 + stream: + eps: 0.001 + capacity: 32 + streamPool: + size: 4096 + samplePool: + size: 4096 + floatsPool: + buckets: + - count: 4096 + capacity: 16 + - count: 2048 + capacity: 32 + - count: 1024 + capacity: 64 + client: + type: m3msg + m3msg: + producer: + writer: + topicName: aggregator_ingest + topicServiceOverride: + zone: embedded + environment: namespace/m3db-cluster-name + placement: + isStaged: true + placementServiceOverride: + namespaces: + placement: /placement + messagePool: + size: 16384 + watermark: + low: 0.2 + high: 0.5 + placementManager: + kvConfig: + namespace: /placement + environment: namespace/m3db-cluster-name + zone: embedded + placementWatcher: + key: m3aggregator + initWatchTimeout: 10s + hashType: murmur32 + bufferDurationBeforeShardCutover: 10m + bufferDurationAfterShardCutoff: 10m + bufferDurationForFutureTimedMetric: 10m # Allow test to write into future. + resignTimeout: 1m + flushTimesManager: + kvConfig: + environment: namespace/m3db-cluster-name + zone: embedded + flushTimesKeyFmt: shardset/%d/flush + flushTimesPersistRetrier: + initialBackoff: 100ms + backoffFactor: 2.0 + maxBackoff: 2s + maxRetries: 3 + electionManager: + election: + leaderTimeout: 10s + resignTimeout: 10s + ttlSeconds: 10 + serviceID: + name: m3aggregator + environment: namespace/m3db-cluster-name + zone: embedded + electionKeyFmt: shardset/%d/lock + campaignRetrier: + initialBackoff: 100ms + backoffFactor: 2.0 + maxBackoff: 2s + forever: true + jitter: true + changeRetrier: + initialBackoff: 100ms + backoffFactor: 2.0 + maxBackoff: 5s + forever: true + jitter: true + resignRetrier: + initialBackoff: 100ms + backoffFactor: 2.0 + maxBackoff: 5s + forever: true + jitter: true + campaignStateCheckInterval: 1s + shardCutoffCheckOffset: 30s + flushManager: + checkEvery: 1s + jitterEnabled: true + maxJitters: + - flushInterval: 5s + maxJitterPercent: 1.0 + - flushInterval: 10s + maxJitterPercent: 0.5 + - flushInterval: 1m + maxJitterPercent: 0.5 + - flushInterval: 10m + maxJitterPercent: 0.5 + - flushInterval: 1h + maxJitterPercent: 0.25 + numWorkersPerCPU: 0.5 + flushTimesPersistEvery: 10s + maxBufferSize: 5m + forcedFlushWindowSize: 10s + flush: + handlers: + - dynamicBackend: + name: m3msg + hashType: murmur32 + producer: + writer: + topicName: aggregated_metrics + topicServiceOverride: + zone: embedded + environment: namespace/m3db-cluster-name + messagePool: + size: 16384 + watermark: + low: 0.2 + high: 0.5 + passthrough: + enabled: true + forwarding: + maxConstDelay: 5m # Need to add some buffer window, since timed metrics by default are delayed by 1min. + entryTTL: 1h + entryCheckInterval: 10m + maxTimerBatchSizePerWrite: 140 + defaultStoragePolicies: [] + maxNumCachedSourceSets: 2 + discardNaNAggregatedValues: true + entryPool: + size: 4096 + counterElemPool: + size: 4096 + timerElemPool: + size: 4096 + gaugeElemPool: + size: 4096 +``` + ## Usage Send metrics as usual to your `m3coordinator` instances in round robin fashion (or any other load balancing strategy), the metrics will be forwarded to the `m3aggregator` instances, then once aggregated they will be returned to the `m3coordinator` instances to write to M3DB. From 622e6c9205b7be3d4dc22454564606b62c1671e0 Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Tue, 4 Aug 2020 20:20:36 -0400 Subject: [PATCH 5/7] Explicitly mentioned aggregated_metrics --- docs/how_to/aggregator.md | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index 2ac3c0b046..3302e43b48 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -54,15 +54,6 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topi }' ``` -#### Initializing m3msg topic for m3coordinator to receive from m3aggregators to write to M3DB - -Now we must setup a topic for the `m3aggregator` to send computed metrics aggregations back to an `m3coordinator` instance for storage to M3DB: -```bash -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic/init -d '{ - "numberOfShards": 64 -}' -``` - #### Add m3aggregagtor consumer group to ingest topic Add the `m3aggregator` placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory retry message buffer): @@ -114,7 +105,7 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST Add the `m3coordinator` placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory retry message buffer): ```bash -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregator_ingest" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ "consumerService": { "serviceId": { "name": "m3coordinator", From bbbfb98ba453b53b675e408534c9a5da48d0fbef Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Tue, 4 Aug 2020 21:26:35 -0400 Subject: [PATCH 6/7] Fix which topic --- docs/how_to/aggregator.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index 3302e43b48..2a76e54669 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -105,7 +105,7 @@ curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -X POST Add the `m3coordinator` placement to receive traffic from the topic (make sure to set message TTL to match your desired maximum in memory retry message buffer): ```bash -curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregator_ingest" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ +curl -vvvsSf -H "Cluster-Environment-Name: namespace/m3db-cluster-name" -H "Topic-Name: aggregated_metrics" -X POST http://m3dbnode-with-embedded-coordinator:7201/api/v1/topic -d '{ "consumerService": { "serviceId": { "name": "m3coordinator", From a7a1b680dfb31d597f205cb161aaec1f554aec2f Mon Sep 17 00:00:00 2001 From: Rob Skillington Date: Tue, 4 Aug 2020 21:36:02 -0400 Subject: [PATCH 7/7] Fixup --- docs/how_to/aggregator.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how_to/aggregator.md b/docs/how_to/aggregator.md index 2a76e54669..8a3881d51a 100644 --- a/docs/how_to/aggregator.md +++ b/docs/how_to/aggregator.md @@ -2,15 +2,15 @@ ## Introduction -`m3aggregator` is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The M3 Coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round round fashion to multiple M3 Coordinators for the same metrics ingestion source (e.g. Prometheus server). +`m3aggregator` is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The M3 Coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round robin fashion to multiple M3 Coordinators for the same metrics ingestion source (e.g. Prometheus server). -Metrics sent to `m3aggregator` are correctly routed to the instance resposible for aggregating each metric and there is multiple m3aggregator replicas to ensure no single point of failure during aggregation. +Similar to M3DB, `m3aggregator` supports clustering and replication by default. This means that metrics are correctly routed to the instance(s) responsible for aggregating each metric and multiple `m3aggregator` replicas can be configured such that there are no single points of failure for aggregation. ## Configuration Before setting up m3aggregator, make sure that you have at least [one M3DB node running](single_node.md) and a dedicated m3coordinator setup. -We highly recommend running with at least a replication factor 2 for a `m3aggregator` deployment. +We highly recommend running with at least a replication factor 2 for a `m3aggregator` deployment. If you run with replication factor 1 then when you restart an aggregator it will temporarily interrupt good the stream of aggregated metrics and there will be some data loss. ### Topology