Implementing dropwizard emitter for druid (fixes #3927)

docs/content/development/extensions-contrib/dropwizard.md

@@ -0,0 +1,100 @@
+---
+layout: doc_page
+---
+
+# Dropwizard Emitter
+
+To use this extension, make sure to [include](../../operations/including-extensions.html) `dropwizard-emitter` extension.
+
+## Introduction
+
+The intent of this extension is to integrate [Dropwizard](http://metrics.dropwizard.io/3.1.0/getting-started/#) metrics library with druid so that dropwizard users can easily absorb druid into their monitoring ecosystem.
+It accumulates druid metrics in a dropwizard histogram and emits them to various sinks via dropwizard supported reporters.
+A histogram measures the statistical distribution of values in a stream of data. In addition to minimum, maximum, mean, etc., it also measures median, 75th, 90th, 95th, 98th, 99th, and 99.9th percentiles.
+Currently dropwizard metrics can be emitted to these sinks:
+Console,HTTP,JMX,Graphite,CSV,Slf4jLogger,Ganglia and various other community supported [sinks](http://metrics.dropwizard.io/3.1.0/manual/third-party/).
+
+## Configuration
+
+All the configuration parameters for Dropwizard emitter are under `druid.emitter.dropwizard`.
+
+|property|description|required?|default|
+|--------|-----------|---------|-------|
+|`druid.emitter.dropwizard.metric`|The metric manager to be used.Currently supported metric manager is histogram|no|histogram|
+|`druid.emitter.dropwizard.reporter`|The dropwizard reporter to be used.|yes|none|
+|`druid.emitter.dropwizard.eventConverter`| Filter and converter of druid events to dropwizard event(please see next section). |yes|none|
+|`druid.emitter.dropwizard.alertEmitters`| List of emitters where alerts will be forwarded to. |no| empty list (no forwarding)|
+
+
+### Druid to Dropwizard Event Converter
+
+Dropwizard Event Converter defines a mapping between druid metrics name plus dimensions to a Dropwizard metric name.
+Dropwizard metric name is organized using the following schema:
+`<namespacePrefix>.[<druid service name>].[<druid hostname>].<druid metrics dimensions>.<druid metrics name>`
+Properly naming the metrics is critical to avoid conflicts, confusing data and potentially wrong interpretation later on.
+
+Example `druid.historical.abc_com:8080.MyDataSourceName.GroupBy.query/time`:
+
+ * `druid` -> namespace prefix 
+ * `historical` -> service name 
+ * `abc.com:8080` -> druid hostname
+ * `MyDataSourceName` -> dimension value 
+ * `GroupBy` -> dimension value
+ * `query/time` -> metric name
+
+We have two different implementation of event converter:
+
+#### Send-All converter
+
+The first implementation called `all`, will send all the druid service metrics events. 
+The metric name will be in the form `<namespacePrefix>.[<druid service name>].[<druid hostname>].<dimensions values ordered by dimension's name>.<metric>`
+User has control of `<namespacePrefix>.[<druid service name>].[<druid hostname>].`
+
+You can omit the hostname by setting `ignoreHostname=true`
+`druid.SERVICE_NAME.dataSourceName.queryType.query.time`
+
+You can omit the service name by setting `ignoreServiceName=true`
+`druid.HOSTNAME.dataSourceName.queryType.query.time`
+
+```json
+
+druid.emitter.dropwizard.eventConverter={"type":"all", "namespacePrefix": "druid.test", "ignoreHostname":true, "ignoreServiceName":true}
+
+```
+
+#### White-list based converter
+
+The second implementation called `whiteList`, will send only the white listed metrics and dimensions.
+Same as for the `all` converter user has control of `<namespacePrefix>.[<druid service name>].[<druid hostname>].`
+White-list based converter comes with the following  default white list map located under resources in `./src/main/resources/defaultWhiteListMap.json`
+
+Although user can override the default white list map by supplying a property called `mapPath`.
+This property is a String containing  the path for the file containing **white list map Json object**.
+For example the following converter will read the map from the file `/pathPrefix/fileName.json`.  
+
+```json
+
+druid.emitter.dropwizard.eventConverter={"type":"whiteList", "namespacePrefix": "druid.test", "ignoreHostname":true, "ignoreServiceName":true, "mapPath":"/pathPrefix/fileName.json"}
+
+```
+
+**Druid emits a huge number of metrics we highly recommend to use the `whiteList` converter**
+
+### Metric Manager
+
+Metric manager defines the dropwizard accumulator that would be used to accumulate druid metric events.
+For eg : Gauge,Counter,Histogram,Meter etc.
+
+### Dropwizard reporter
+
+```json
+
+druid.emitter.dropwizard.reporter={"type":"jmx"}
+
+```
+
+```json
+
+druid.emitter.dropwizard.reporter={"type":"console","emitIntervalInSecs":30}"}
+
+```

extensions-contrib/dropwizard-emitter/pom.xml

@@ -0,0 +1,91 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to Metamarkets Group Inc. (Metamarkets) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. Metamarkets licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <parent>
+    <groupId>io.druid</groupId>
+    <artifactId>druid</artifactId>
+    <version>0.10.0-SNAPSHOT</version>
+    <relativePath>../../pom.xml</relativePath>
+  </parent>
+
+  <groupId>io.druid.extensions.contrib</groupId>
+  <artifactId>dropwizard-emitter</artifactId>
+  <name>dropwizard-emitter</name>
+  <description>Druid emitter extension to convert druid metric to Dropwizard metrics</description>
+
+  <dependencies>
+    <dependency>
+      <groupId>io.druid</groupId>
+      <artifactId>druid-common</artifactId>
+      <version>${project.parent.version}</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.druid</groupId>
+      <artifactId>druid-api</artifactId>
+      <version>${project.parent.version}</version>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>com.metamx</groupId>
+      <artifactId>emitter</artifactId>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>junit</groupId>
+      <artifactId>junit</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.easymock</groupId>
+      <artifactId>easymock</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>pl.pragmatists</groupId>
+      <artifactId>JUnitParams</artifactId>
+      <version>1.0.4</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.druid</groupId>
+      <artifactId>druid-server</artifactId>
+      <version>${project.parent.version}</version>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.druid</groupId>
+      <artifactId>druid-processing</artifactId>
+      <version>${project.parent.version}</version>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+      <dependency>
+          <groupId>io.dropwizard.metrics</groupId>
+          <artifactId>metrics-core</artifactId>
+          <version>3.1.0</version>
+      </dependency>
+  </dependencies>
+</project>

extensions-contrib/dropwizard-emitter/src/main/java/io/druid/emitter/dropwizard/DropwizardEmitter.java

@@ -0,0 +1,108 @@
+/*
+ * Licensed to Metamarkets Group Inc. (Metamarkets) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. Metamarkets licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package io.druid.emitter.dropwizard;
+
+import com.codahale.metrics.MetricRegistry;
+import com.metamx.emitter.core.Emitter;
+import com.metamx.emitter.core.Event;
+import com.metamx.emitter.service.AlertEvent;
+import com.metamx.emitter.service.ServiceMetricEvent;
+import io.druid.java.util.common.ISE;
+import io.druid.java.util.common.logger.Logger;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.regex.Pattern;
+
+public class DropwizardEmitter implements Emitter {
+    private static Logger log = new Logger(DropwizardEmitter.class);
+    private final MetricRegistry metricsRegistry = new MetricRegistry();
+    private final DropwizardMetricManager dropwizardMetricManager;
+    private final AtomicBoolean started = new AtomicBoolean(false);
+    private final DruidToDropwizardEventConverter druidToDropwizardEventConverter;
+    private final List<Emitter> emitterList;
+    //TODO make this a list of reporters
+    private final DropwizardReporter dropwizardReporter;
+
+    public DropwizardEmitter(DropwizardEmitterConfig dropwizardEmitterConfig,List<Emitter> emitterList) {
+        this.dropwizardMetricManager = dropwizardEmitterConfig.getDropwizardMetricManager();
+        this.druidToDropwizardEventConverter = dropwizardEmitterConfig.getDruidToDropwizardEventConverter();
+        this.emitterList = emitterList;
+        this.dropwizardReporter = dropwizardEmitterConfig.getDropwizardReporter();
+    }
+
+    @Override
+    public void start() {
+        try {
+            startReporters();
+        } catch (IOException e) {
+            log.error(e,"Error while starting Dropwizard reporters");
+        }
+        started.set(true);
+    }
+
+    @Override
+    public void emit(Event event) {
+        if (!started.get()) {
+            throw new ISE("Emit was called while emitter is yet not initialized");
+        }
+        if (event instanceof ServiceMetricEvent) {
+            DropwizardEvent dropwizardEvent = druidToDropwizardEventConverter.druidEventToDropwizard((ServiceMetricEvent) event);
+            if(dropwizardEvent!=null) {
+                dropwizardMetricManager.updateMetric(metricsRegistry,dropwizardEvent);
+            }else{
+                log.debug("Dropping the service event "+event);
+                return;
+            }
+        } else if (!emitterList.isEmpty() && event instanceof AlertEvent) {
+            for (Emitter emitter : emitterList) {
+                emitter.emit(event);
+            }
+        } else if (event instanceof AlertEvent) {
+            AlertEvent alertEvent = (AlertEvent) event;
+            log.error(
+                    "The following alert is dropped, description is [%s], severity is [%s]",
+                    alertEvent.getDescription(), alertEvent.getSeverity()
+            );
+        } else {
+            log.error("unknown event type [%s]", event.getClass());
+        }
+    }
+
+    @Override
+    public void flush() throws IOException {
+        dropwizardReporter.flush();
+    }
+
+    @Override
+    public void close() throws IOException {
+        dropwizardReporter.close();
+    }
+
+    private void startReporters() throws IOException {
+        dropwizardReporter.start(metricsRegistry);
+    }
+
+    public static String sanitize(String namespace)
+    {
+        Pattern DOT_OR_WHITESPACE = Pattern.compile("[\\s]+|[.]+");
+        return DOT_OR_WHITESPACE.matcher(namespace).replaceAll("_");
+    }
+}

extensions-contrib/dropwizard-emitter/src/main/java/io/druid/emitter/dropwizard/DropwizardEmitterConfig.java

@@ -0,0 +1,91 @@
+/*
+ * Licensed to Metamarkets Group Inc. (Metamarkets) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. Metamarkets licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package io.druid.emitter.dropwizard;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.Collections;
+import java.util.List;
+
+
+public class DropwizardEmitterConfig {
+    @JsonProperty("eventConverter")
+    final private DruidToDropwizardEventConverter druidToDropwizardEventConverter;
+    @JsonProperty("reporter")
+    final private DropwizardReporter dropwizardReporter;
+    @JsonProperty("metric")
+    final private DropwizardMetricManager dropwizardMetricManager;
+    @JsonProperty
+    final private List<String> alertEmitters;
+
+    @JsonCreator
+    public DropwizardEmitterConfig(@JsonProperty("reporter") DropwizardReporter dropwizardReporter ,@JsonProperty("eventConverter") DruidToDropwizardEventConverter druidToDropwizardEventConverter,@JsonProperty("metric") DropwizardMetricManager dropwizardMetricManager,@JsonProperty("alertEmitters") List<String> alertEmitters ) {
+        this.dropwizardReporter = dropwizardReporter;
+        this.druidToDropwizardEventConverter = druidToDropwizardEventConverter;
+        this.dropwizardMetricManager = dropwizardMetricManager==null? new HistogramMetricManager():dropwizardMetricManager;
+        this.alertEmitters = alertEmitters == null ? Collections.<String>emptyList() : alertEmitters;
+    }
+
+
+    public DruidToDropwizardEventConverter getDruidToDropwizardEventConverter() {
+        return druidToDropwizardEventConverter;
+    }
+
+    public DropwizardReporter getDropwizardReporter() {
+        return dropwizardReporter;
+    }
+
+    public List<String> getAlertEmitters() {
+        return alertEmitters;
+    }
+
+    public DropwizardMetricManager getDropwizardMetricManager() {
+        return dropwizardMetricManager;
+    }
+
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) return true;
+        if (o == null || getClass() != o.getClass()) return false;
+
+        DropwizardEmitterConfig that = (DropwizardEmitterConfig) o;
+
+        if (alertEmitters != null ? !alertEmitters.equals(that.alertEmitters) : that.alertEmitters != null)
+            return false;
+        if (dropwizardMetricManager != null ? !dropwizardMetricManager.equals(that.dropwizardMetricManager) : that.dropwizardMetricManager != null)
+            return false;
+        if (dropwizardReporter != null ? !dropwizardReporter.equals(that.dropwizardReporter) : that.dropwizardReporter != null)
+            return false;
+        if (druidToDropwizardEventConverter != null ? !druidToDropwizardEventConverter.equals(that.druidToDropwizardEventConverter) : that.druidToDropwizardEventConverter != null)
+            return false;
+
+        return true;
+    }
+
+    @Override
+    public int hashCode() {
+        int result = druidToDropwizardEventConverter != null ? druidToDropwizardEventConverter.hashCode() : 0;
+        result = 31 * result + (dropwizardReporter != null ? dropwizardReporter.hashCode() : 0);
+        result = 31 * result + (dropwizardMetricManager != null ? dropwizardMetricManager.hashCode() : 0);
+        result = 31 * result + (alertEmitters != null ? alertEmitters.hashCode() : 0);
+        return result;
+    }
+}