From 6fddb4361535c0d07845509fb0196c123fccf128 Mon Sep 17 00:00:00 2001 From: ZhangJian He Date: Wed, 13 Jul 2022 23:45:08 +0800 Subject: [PATCH] Bump prometheus client version from 0.5.0 to 0.15.0 (#13785) ### Motivation - prometheus `simpleclient` has broken api, it makes `pulsar-client-origin` can not well used with other libraries - specify `OpenMetrics` This PR contains metric name broken changes. ### Modifications Bump prometheus client version from 0.5.0 to 0.15.0 ### changes - bump the client version - rename the field prometheus client `UNTYPED` to `UNKNOWN` - `OpenMetrics`'s counter name needs a `_total` suffix ### metrics name broken changes - rename `pulsar_expired_token_count` to `pulsar_expired_token_total` - rename `pulsar_authentication_success_count` to `pulsar_authentication_success_total` - rename `pulsar_authentication_failures_count` to `pulsar_authentication_failures_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_source_received_total_1min` to `pulsar_source_received_1min_total` - rename `pulsar_source_written_total_1min` to `pulsar_source_written_1min_total` - rename `pulsar_source_source_total_1min` to `pulsar_source_source_exceptions_1min_total ` - rename `pulsar_source_system_exceptions_total_1min` to `pulsar_source_system_exceptions_1min_total ` - rename `pulsar_function_received_total_1min` to `pulsar_function_received_1min_total ` - rename `pulsar_function_user_exceptions_total_1min` to `pulsar_function_user_exceptions_1min_total` - rename `pulsar_function_system_exceptions_total_1min` to `pulsar_function_system_exceptions_1min_total` - rename `pulsar_function_processed_successfully_total_1min` to `pulsar_function_processed_successfully_1min_total` - rename `pulsar_function_received_total_1min` to `pulsar_function_received_1min_total` - rename `pulsar_function_user_exceptions_total_1min` to `pulsar_function_user_exceptions_1min_total` - rename `pulsar_function_system_exceptions_total_1min` to `pulsar_function_system_exceptions_1min_total` - rename `pulsar_function_processed_successfully_total_1min` to `pulsar_function_processed_successfully_1min_total` - rename `pulsar_sink_received_total_1min` to `pulsar_sink_received_1min_total` - rename `pulsar_sink_written_total_1min` to `pulsar_sink_written_1min_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_sink_system_exceptions_total_1min` to `pulsar_sink_system_exceptions_1min_total` - rename `pulsar_sink_received_total_1min` to `pulsar_sink_received_1min_total` - rename `pulsar_sink_written_total_1min` to `pulsar_sink_written_1min_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_sink_system_exceptions_total_1min` to `pulsar_sink_system_exceptions_1min_total` ### Documentation Check the box below or label this PR directly. Need to update docs? - [x] `doc-required` As mentioned above, the metrics name has changed --- .../server/src/assemble/LICENSE.bin.txt | 23 +++++++------ pom.xml | 4 +-- .../AuthenticationProviderToken.java | 2 +- .../metrics/AuthenticationMetrics.java | 4 +-- .../PrometheusMetricsGeneratorUtils.java | 16 +++++++--- .../broker/stats/PrometheusMetricsTest.java | 16 ++++++++-- .../worker/PulsarFunctionLocalRunTest.java | 2 +- .../pulsar/io/PulsarBatchSourceE2ETest.java | 8 ++--- .../pulsar/io/PulsarFunctionE2ETest.java | 16 +++++----- .../apache/pulsar/io/PulsarSinkE2ETest.java | 16 +++++----- .../apache/pulsar/io/PulsarSourceE2ETest.java | 8 ++--- .../resources/prometheus_metrics_sample.txt | 16 +++++----- .../instance/stats/FunctionStatsManager.java | 12 +++---- .../instance/stats/SinkStatsManager.java | 8 ++--- .../instance/stats/SourceStatsManager.java | 8 ++--- .../src/main/python/function_stats.py | 8 ++--- .../server/ProxyPrometheusMetricsTest.java | 4 +-- pulsar-sql/presto-distribution/LICENSE | 12 ++++--- site2/docs/reference-metrics.md | 32 +++++++++---------- 19 files changed, 121 insertions(+), 94 deletions(-) diff --git a/distribution/server/src/assemble/LICENSE.bin.txt b/distribution/server/src/assemble/LICENSE.bin.txt index 34a3ca422b081e..8f0b138d4d832d 100644 --- a/distribution/server/src/assemble/LICENSE.bin.txt +++ b/distribution/server/src/assemble/LICENSE.bin.txt @@ -377,14 +377,19 @@ The Apache Software License, Version 2.0 - io.netty-netty-tcnative-boringssl-static-2.0.52.Final-windows-x86_64.jar - io.netty-netty-tcnative-classes-2.0.52.Final.jar * Prometheus client - - io.prometheus-simpleclient-0.5.0.jar - - io.prometheus-simpleclient_common-0.5.0.jar - - io.prometheus-simpleclient_hotspot-0.5.0.jar - - io.prometheus-simpleclient_servlet-0.5.0.jar - - io.prometheus-simpleclient_log4j2-0.5.0.jar - - io.prometheus-simpleclient_jetty-0.5.0.jar - - io.prometheus.jmx-collector-0.14.0.jar - - io.prometheus-simpleclient_caffeine-0.5.0.jar + - io.prometheus.jmx-collector-0.16.1.jar + - io.prometheus-simpleclient-0.15.0.jar + - io.prometheus-simpleclient_caffeine-0.15.0.jar + - io.prometheus-simpleclient_common-0.15.0.jar + - io.prometheus-simpleclient_hotspot-0.15.0.jar + - io.prometheus-simpleclient_httpserver-0.15.0.jar + - io.prometheus-simpleclient_jetty-0.15.0.jar + - io.prometheus-simpleclient_log4j2-0.15.0.jar + - io.prometheus-simpleclient_servlet-0.15.0.jar + - io.prometheus-simpleclient_servlet_common-0.15.0.jar + - io.prometheus-simpleclient_tracer_common-0.15.0.jar + - io.prometheus-simpleclient_tracer_otel-0.15.0.jar + - io.prometheus-simpleclient_tracer_otel_agent-0.15.0.jar * Jakarta Bean Validation API - jakarta.validation-jakarta.validation-api-2.0.2.jar - javax.validation-validation-api-1.1.0.Final.jar @@ -508,7 +513,7 @@ The Apache Software License, Version 2.0 - io.dropwizard.metrics-metrics-jvm-4.1.12.1.jar - io.dropwizard.metrics-metrics-jmx-4.1.12.1.jar * Prometheus - - io.prometheus-simpleclient_httpserver-0.5.0.jar + - io.prometheus-simpleclient_httpserver-0.15.0.jar * Java JSON WebTokens - io.jsonwebtoken-jjwt-api-0.11.1.jar - io.jsonwebtoken-jjwt-impl-0.11.1.jar diff --git a/pom.xml b/pom.xml index 50a7217da719f4..cb35cf9ce7ccda 100644 --- a/pom.xml +++ b/pom.xml @@ -126,7 +126,7 @@ flexible messaging model and an intuitive client API. 2.5.2 2.34 1.10.50 - 0.5.0 + 0.15.0 3.9.8 6.29.4.1 1.7.32 @@ -179,7 +179,7 @@ flexible messaging model and an intuitive client API. 2.4.9 31.0.1-jre 1.0 - 0.14.0 + 0.16.1 7.0.1 5.3.0 5.3.0 diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java index ccd4e8e1aaba2c..5c6d17a1d776b9 100644 --- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java +++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/AuthenticationProviderToken.java @@ -74,7 +74,7 @@ public class AuthenticationProviderToken implements AuthenticationProvider { static final String TOKEN = "token"; private static final Counter expiredTokenMetrics = Counter.build() - .name("pulsar_expired_token_count") + .name("pulsar_expired_token_total") .help("Pulsar expired token") .register(); diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/metrics/AuthenticationMetrics.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/metrics/AuthenticationMetrics.java index 60565db521d892..b4ae44e200d010 100644 --- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/metrics/AuthenticationMetrics.java +++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/authentication/metrics/AuthenticationMetrics.java @@ -22,12 +22,12 @@ public class AuthenticationMetrics { private static final Counter authSuccessMetrics = Counter.build() - .name("pulsar_authentication_success_count") + .name("pulsar_authentication_success_total") .help("Pulsar authentication success") .labelNames("provider_name", "auth_method") .register(); private static final Counter authFailuresMetrics = Counter.build() - .name("pulsar_authentication_failures_count") + .name("pulsar_authentication_failures_total") .help("Pulsar authentication failures") .labelNames("provider_name", "auth_method", "reason") .register(); diff --git a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java index ead3c332b2be3c..d341e699bf7e81 100644 --- a/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java +++ b/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/stats/prometheus/PrometheusMetricsGeneratorUtils.java @@ -59,7 +59,7 @@ public static void generateSystemMetrics(SimpleTextOutputStream stream, String c Collector.MetricFamilySamples metricFamily = metricFamilySamples.nextElement(); // Write type of metric - stream.write("# TYPE ").write(metricFamily.name).write(' ') + stream.write("# TYPE ").write(metricFamily.name).write(getTypeNameSuffix(metricFamily.type)).write(' ') .write(getTypeStr(metricFamily.type)).write('\n'); for (int i = 0; i < metricFamily.samples.size(); i++) { @@ -88,19 +88,27 @@ public static void generateSystemMetrics(SimpleTextOutputStream stream, String c } } + static String getTypeNameSuffix(Collector.Type type) { + if (type.equals(Collector.Type.INFO)) { + return "_info"; + } + return ""; + } + static String getTypeStr(Collector.Type type) { switch (type) { case COUNTER: return "counter"; case GAUGE: + case INFO: return "gauge"; - case SUMMARY : + case SUMMARY: return "summary"; case HISTOGRAM: return "histogram"; - case UNTYPED: + case UNKNOWN: default: - return "untyped"; + return "unknown"; } } diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java index 6663410e1181f5..8eccf1785e4bf1 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/broker/stats/PrometheusMetricsTest.java @@ -847,6 +847,16 @@ public void testDuplicateMetricTypeDefinitions() throws Exception { if (!typeDefs.containsKey(summaryMetricName)) { fail("Metric " + metricName + " does not have a corresponding summary type definition"); } + } else if (metricName.endsWith("_created")) { + String summaryMetricName = metricName.substring(0, metricName.indexOf("_created")); + if (!typeDefs.containsKey(summaryMetricName)) { + fail("Metric " + metricName + " does not have a corresponding summary type definition"); + } + } else if (metricName.endsWith("_total")) { + String summaryMetricName = metricName.substring(0, metricName.indexOf("_total")); + if (!typeDefs.containsKey(summaryMetricName)) { + fail("Metric " + metricName + " does not have a corresponding counter type definition"); + } } else { fail("Metric " + metricName + " does not have a type definition"); } @@ -1087,7 +1097,7 @@ public String getCommandData() { PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut); String metricsStr = statsOut.toString(); Multimap metrics = parseMetrics(metricsStr); - List cm = (List) metrics.get("pulsar_authentication_success_count"); + List cm = (List) metrics.get("pulsar_authentication_success_total"); boolean haveSucceed = false; for (Metric metric : cm) { if (Objects.equals(metric.tags.get("auth_method"), "token") @@ -1097,7 +1107,7 @@ public String getCommandData() { } Assert.assertTrue(haveSucceed); - cm = (List) metrics.get("pulsar_authentication_failures_count"); + cm = (List) metrics.get("pulsar_authentication_failures_total"); boolean haveFailed = false; for (Metric metric : cm) { @@ -1147,7 +1157,7 @@ public String getCommandData() { PrometheusMetricsGenerator.generate(pulsar, false, false, false, statsOut); String metricsStr = statsOut.toString(); Multimap metrics = parseMetrics(metricsStr); - List cm = (List) metrics.get("pulsar_expired_token_count"); + List cm = (List) metrics.get("pulsar_expired_token_total"); assertEquals(cm.size(), 1); provider.close(); diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java index a8344be8dc8718..e810a1d7d93fb7 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/functions/worker/PulsarFunctionLocalRunTest.java @@ -987,7 +987,7 @@ private void testPulsarSinkLocalRun(String jarFilePathUrl, int parallelism, Stri assertFalse(metrics.isEmpty()); PulsarFunctionTestUtils.Metric m = metrics.get("pulsar_sink_sink_exceptions_total"); if (m == null) { - m = metrics.get("pulsar_sink_sink_exceptions_total_1min"); + m = metrics.get("pulsar_sink_sink_exceptions_1min_total"); } assertEquals(m.value, 0); } diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarBatchSourceE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarBatchSourceE2ETest.java index d1cd1dc0614e3c..e371190a0d0e5c 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarBatchSourceE2ETest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarBatchSourceE2ETest.java @@ -113,7 +113,7 @@ private void testPulsarBatchSourceStats(String jarFilePathUrl) throws Exception assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertTrue(m.value > 0.0); - m = metrics.get("pulsar_source_received_total_1min"); + m = metrics.get("pulsar_source_received_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -127,7 +127,7 @@ private void testPulsarBatchSourceStats(String jarFilePathUrl) throws Exception assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertTrue(m.value > 0.0); - m = metrics.get("pulsar_source_written_total_1min"); + m = metrics.get("pulsar_source_written_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -141,7 +141,7 @@ private void testPulsarBatchSourceStats(String jarFilePathUrl) throws Exception assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_source_source_exceptions_total_1min"); + m = metrics.get("pulsar_source_source_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -155,7 +155,7 @@ private void testPulsarBatchSourceStats(String jarFilePathUrl) throws Exception assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_source_system_exceptions_total_1min"); + m = metrics.get("pulsar_source_system_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java index 32ea853f2690fb..1502862d676944 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java @@ -384,7 +384,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_received_total_1min"); + m = metrics.get("pulsar_function_received_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -398,7 +398,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_user_exceptions_total_1min"); + m = metrics.get("pulsar_function_user_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -426,7 +426,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_system_exceptions_total_1min"); + m = metrics.get("pulsar_function_system_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -447,7 +447,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_processed_successfully_total_1min"); + m = metrics.get("pulsar_function_processed_successfully_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -544,7 +544,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, (double) totalMsgs); - m = metrics.get("pulsar_function_received_total_1min"); + m = metrics.get("pulsar_function_received_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -558,7 +558,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_user_exceptions_total_1min"); + m = metrics.get("pulsar_function_user_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -586,7 +586,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_function_system_exceptions_total_1min"); + m = metrics.get("pulsar_function_system_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); @@ -607,7 +607,7 @@ public void testPulsarFunctionStats() throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, functionName)); assertEquals(m.value, (double) totalMsgs); - m = metrics.get("pulsar_function_processed_successfully_total_1min"); + m = metrics.get("pulsar_function_processed_successfully_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), functionName); diff --git a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java index 4b7578dc606565..24bea1d59940c9 100644 --- a/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java +++ b/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarSinkE2ETest.java @@ -284,7 +284,7 @@ private void testPulsarSinkStats(String jarFilePathUrl, Function 0.0); - m = metrics.get("pulsar_source_received_total_1min"); + m = metrics.get("pulsar_source_received_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -131,7 +131,7 @@ private void testPulsarSourceStats(String jarFilePathUrl) throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertTrue(m.value > 0.0); - m = metrics.get("pulsar_source_written_total_1min"); + m = metrics.get("pulsar_source_written_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -145,7 +145,7 @@ private void testPulsarSourceStats(String jarFilePathUrl) throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_source_source_exceptions_total_1min"); + m = metrics.get("pulsar_source_source_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); @@ -159,7 +159,7 @@ private void testPulsarSourceStats(String jarFilePathUrl) throws Exception { assertEquals(m.tags.get("namespace"), String.format("%s/%s", tenant, namespacePortion)); assertEquals(m.tags.get("fqfn"), FunctionCommon.getFullyQualifiedName(tenant, namespacePortion, sourceName)); assertEquals(m.value, 0.0); - m = metrics.get("pulsar_source_system_exceptions_total_1min"); + m = metrics.get("pulsar_source_system_exceptions_1min_total"); assertEquals(m.tags.get("cluster"), config.getClusterName()); assertEquals(m.tags.get("instance_id"), "0"); assertEquals(m.tags.get("name"), sourceName); diff --git a/pulsar-broker/src/test/resources/prometheus_metrics_sample.txt b/pulsar-broker/src/test/resources/prometheus_metrics_sample.txt index 20c453c1f4c064..5df34377d107ae 100644 --- a/pulsar-broker/src/test/resources/prometheus_metrics_sample.txt +++ b/pulsar-broker/src/test/resources/prometheus_metrics_sample.txt @@ -83,8 +83,8 @@ jvm_buffer_pool_used_buffers{cluster="use",pool="direct"} 97.0 jvm_buffer_pool_used_buffers{cluster="use",pool="mapped"} 0.0 # TYPE pulsar_broker_lookup_pending_requests gauge pulsar_broker_lookup_pending_requests{cluster="use"} 0.0 -# TYPE pulsar_authentication_success_count counter -pulsar_authentication_success_count{cluster="use",provider_name="AuthenticationProviderTls",auth_method="tls"} 850.0 +# TYPE pulsar_authentication_success_total counter +pulsar_authentication_success_total{cluster="use",provider_name="AuthenticationProviderTls",auth_method="tls"} 850.0 # TYPE pulsar_version_info gauge pulsar_version_info{cluster="use",version="2.8.0-SNAPSHOT",commit="e600b65a05e610bc7cbd874d4c446619d9d9606f"} 1.0 # TYPE zk_read_latency summary @@ -311,8 +311,8 @@ jvm_gc_collection_seconds_count{cluster="use",gc="G1 Old Generation"} 0.0 jvm_gc_collection_seconds_sum{cluster="use",gc="G1 Old Generation"} 0.0 # TYPE pulsar_broker_lookup_failures counter pulsar_broker_lookup_failures{cluster="use"} 0.0 -# TYPE pulsar_authentication_failures_count counter -pulsar_authentication_failures_count{cluster="use",provider_name="AuthenticationProviderTls",auth_method="tls",reason="Client unable to authenticate with TLS certificate"} 1.0 +# TYPE pulsar_authentication_failures_total counter +pulsar_authentication_failures_total{cluster="use",provider_name="AuthenticationProviderTls",auth_method="tls",reason="Client unable to authenticate with TLS certificate"} 1.0 # TYPE pulsar_broker_lookup_answers counter pulsar_broker_lookup_answers{cluster="use"} 134.0 # TYPE pulsar_topics_count gauge @@ -665,8 +665,8 @@ pulsar_function_worker_total_function_count{cluster="use",} 1 pulsar_function_worker_total_expected_instance_count{cluster="use",} 1 pulsar_function_worker_is_leader{cluster="use",} 1 pulsar_function_last_invocation{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 -pulsar_function_processed_successfully_total_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 -pulsar_function_received_total_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 +pulsar_function_processed_successfully_1min_total{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 +pulsar_function_received_1min_total{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 pulsar_function_user_exceptions_total{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 pulsar_function_process_latency_ms_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",quantile="0.5",} NaN pulsar_function_process_latency_ms_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",quantile="0.9",} NaN @@ -683,8 +683,8 @@ pulsar_function_process_latency_ms{tenant="external-repl-prop",namespace="extern pulsar_function_process_latency_ms{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",quantile="0.999",} NaN pulsar_function_process_latency_ms_count{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 pulsar_function_process_latency_ms_sum{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 -pulsar_function_system_exceptions_total_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 -pulsar_function_user_exceptions_total_1min{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 +pulsar_function_system_exceptions_1min_total{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 +pulsar_function_user_exceptions_1min_total{tenant="external-repl-prop",namespace="external-repl-prop/io",name="PulsarSink-test",instance_id="0",cluster="use",fqfn="external-repl-prop/io/PulsarSink-test",} 0.0 # TYPE pulsar_ml_cache_evictions gauge pulsar_ml_cache_evictions{cluster="use"} 0 1617950344971 # TYPE pulsar_ml_cache_hits_rate gauge diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java index f693b2778c2aab..c15262fb0f6839 100644 --- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java +++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java @@ -51,13 +51,13 @@ public class FunctionStatsManager extends ComponentStatsManager { public static final String LAST_INVOCATION = "last_invocation"; public static final String RECEIVED_TOTAL = "received_total"; - public static final String PROCESSED_SUCCESSFULLY_TOTAL_1min = "processed_successfully_total_1min"; - public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_total_1min"; - public static final String USER_EXCEPTIONS_TOTAL_1min = "user_exceptions_total_1min"; - public static final String SOURCE_EXCEPTIONS_TOTAL_1min = "source_exceptions_total_1min"; - public static final String SINK_EXCEPTIONS_TOTAL_1min = "sink_exceptions_total_1min"; + public static final String PROCESSED_SUCCESSFULLY_TOTAL_1min = "processed_successfully_1min"; + public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_1min"; + public static final String USER_EXCEPTIONS_TOTAL_1min = "user_exceptions_1min"; + public static final String SOURCE_EXCEPTIONS_TOTAL_1min = "source_exceptions_1min"; + public static final String SINK_EXCEPTIONS_TOTAL_1min = "sink_exceptions_1min"; public static final String PROCESS_LATENCY_MS_1min = "process_latency_ms_1min"; - public static final String RECEIVED_TOTAL_1min = "received_total_1min"; + public static final String RECEIVED_TOTAL_1min = "received_1min"; /** Declare Prometheus stats. **/ diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SinkStatsManager.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SinkStatsManager.java index 4f0bbb04707abc..10460dcc8b0462 100644 --- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SinkStatsManager.java +++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SinkStatsManager.java @@ -39,10 +39,10 @@ public class SinkStatsManager extends ComponentStatsManager { public static final String RECEIVED_TOTAL = "received_total"; public static final String WRITTEN_TOTAL = "written_total"; - public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_total_1min"; - public static final String SINK_EXCEPTIONS_TOTAL_1min = "sink_exceptions_total_1min"; - public static final String RECEIVED_TOTAL_1min = "received_total_1min"; - public static final String WRITTEN_TOTAL_1min = "written_total_1min"; + public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_1min"; + public static final String SINK_EXCEPTIONS_TOTAL_1min = "sink_exceptions_1min"; + public static final String RECEIVED_TOTAL_1min = "received_1min"; + public static final String WRITTEN_TOTAL_1min = "written_1min"; /** Declare Prometheus stats. **/ diff --git a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SourceStatsManager.java b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SourceStatsManager.java index e050dcb13dc59f..ad470aaa2ed9b6 100644 --- a/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SourceStatsManager.java +++ b/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/SourceStatsManager.java @@ -39,10 +39,10 @@ public class SourceStatsManager extends ComponentStatsManager { public static final String RECEIVED_TOTAL = "received_total"; public static final String WRITTEN_TOTAL = "written_total"; - public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_total_1min"; - public static final String SOURCE_EXCEPTIONS_TOTAL_1min = "source_exceptions_total_1min"; - public static final String RECEIVED_TOTAL_1min = "received_total_1min"; - public static final String WRITTEN_TOTAL_1min = "written_total_1min"; + public static final String SYSTEM_EXCEPTIONS_TOTAL_1min = "system_exceptions_1min"; + public static final String SOURCE_EXCEPTIONS_TOTAL_1min = "source_exceptions_1min"; + public static final String RECEIVED_TOTAL_1min = "received_1min"; + public static final String WRITTEN_TOTAL_1min = "written_1min"; /** Declare Prometheus stats. **/ diff --git a/pulsar-functions/instance/src/main/python/function_stats.py b/pulsar-functions/instance/src/main/python/function_stats.py index 8b54f75044de21..39246a83c47689 100644 --- a/pulsar-functions/instance/src/main/python/function_stats.py +++ b/pulsar-functions/instance/src/main/python/function_stats.py @@ -41,11 +41,11 @@ class Stats(object): LAST_INVOCATION = 'last_invocation' TOTAL_RECEIVED = 'received_total' - TOTAL_SUCCESSFULLY_PROCESSED_1min = 'processed_successfully_total_1min' - TOTAL_SYSTEM_EXCEPTIONS_1min = 'system_exceptions_total_1min' - TOTAL_USER_EXCEPTIONS_1min = 'user_exceptions_total_1min' + TOTAL_SUCCESSFULLY_PROCESSED_1min = 'processed_successfully_1min_total' + TOTAL_SYSTEM_EXCEPTIONS_1min = 'system_exceptions_1min_total' + TOTAL_USER_EXCEPTIONS_1min = 'user_exceptions_1min_total' PROCESS_LATENCY_MS_1min = 'process_latency_ms_1min' - TOTAL_RECEIVED_1min = 'received_total_1min' + TOTAL_RECEIVED_1min = 'received_1min_total' # Declare Prometheus stat_total_processed_successfully = Counter(PULSAR_FUNCTION_METRICS_PREFIX + TOTAL_SUCCESSFULLY_PROCESSED, diff --git a/pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyPrometheusMetricsTest.java b/pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyPrometheusMetricsTest.java index 86e0b8727ab61a..6d7ba4c72df73d 100644 --- a/pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyPrometheusMetricsTest.java +++ b/pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/ProxyPrometheusMetricsTest.java @@ -107,7 +107,7 @@ public void testMetrics() { Multimap metrics = parseMetrics(response); // Check that ProxyService metrics are present - List cm = (List) metrics.get("pulsar_proxy_binary_bytes"); + List cm = (List) metrics.get("pulsar_proxy_binary_bytes_total"); assertEquals(cm.size(), 1); assertEquals(cm.get(0).tags.get("cluster"), TEST_CLUSTER); @@ -117,7 +117,7 @@ public void testMetrics() { assertEquals(cm2.get(0).tags.get("label1"), "xyz"); // Check that PrometheusRawMetricsProvider metrics are present - List cm3 = (List) metrics.get("test_counter"); + List cm3 = (List) metrics.get("test_counter_total"); assertEquals(cm3.size(), 1); assertEquals(cm3.get(0).tags.get("cluster"), TEST_CLUSTER); } diff --git a/pulsar-sql/presto-distribution/LICENSE b/pulsar-sql/presto-distribution/LICENSE index cd19bb308c769f..32880122173ce4 100644 --- a/pulsar-sql/presto-distribution/LICENSE +++ b/pulsar-sql/presto-distribution/LICENSE @@ -419,10 +419,14 @@ The Apache Software License, Version 2.0 - metrics-graphite-4.1.12.1.jar - metrics-jvm-4.1.12.1.jar * Prometheus - - simpleclient-0.5.0.jar - - simpleclient_common-0.5.0.jar - - simpleclient_hotspot-0.5.0.jar - - simpleclient_servlet-0.5.0.jar + - simpleclient-0.15.0.jar + - simpleclient_common-0.15.0.jar + - simpleclient_hotspot-0.15.0.jar + - simpleclient_servlet-0.15.0.jar + - simpleclient_servlet_common-0.15.0.jar + - simpleclient_tracer_common-0.15.0.jar + - simpleclient_tracer_otel-0.15.0.jar + - simpleclient_tracer_otel_agent-0.15.0.jar * JCTools - jctools-core-2.1.2.jar * Asynchronous Http Client diff --git a/site2/docs/reference-metrics.md b/site2/docs/reference-metrics.md index 3ee6aa0e52b564..b7493367c7d79f 100644 --- a/site2/docs/reference-metrics.md +++ b/site2/docs/reference-metrics.md @@ -479,7 +479,7 @@ All the token metrics are labelled with the following labels: | Name | Type | Description | |---|---|---| -| pulsar_expired_token_count | Counter | The number of expired tokens in Pulsar. | +| pulsar_expired_token_total | Counter | The number of expired tokens in Pulsar. | | pulsar_expiring_token_minutes | Histogram | The remaining time of expiring tokens in minutes. | ### Authentication metrics @@ -489,12 +489,12 @@ All the authentication metrics are labelled with the following labels: - *cluster*: `cluster=${pulsar_cluster}`. `${pulsar_cluster}` is the cluster name that you have configured in the `broker.conf` file. - *provider_name*: `provider_name=${provider_name}`. `${provider_name}` is the class name of the authentication provider. - *auth_method*: `auth_method=${auth_method}`. `${auth_method}` is the authentication method of the authentication provider. -- *reason*: `reason=${reason}`. `${reason}` is the reason for failing authentication operation. (This label is only for `pulsar_authentication_failures_count`.) +- *reason*: `reason=${reason}`. `${reason}` is the reason for failing authentication operation. (This label is only for `pulsar_authentication_failures_total`.) | Name | Type | Description | |---|---|---| -| pulsar_authentication_success_count| Counter | The number of successful authentication operations. | -| pulsar_authentication_failures_count | Counter | The number of failing authentication operations. | +| pulsar_authentication_success_total| Counter | The number of successful authentication operations. | +| pulsar_authentication_failures_total | Counter | The number of failing authentication operations. | ### Jetty metrics @@ -591,16 +591,16 @@ All the Pulsar Functions metrics are labelled with the following labels: | Name | Type | Description | |---|---|---| | pulsar_function_processed_successfully_total | Counter | The total number of messages processed successfully. | -| pulsar_function_processed_successfully_total_1min | Counter | The total number of messages processed successfully in the last 1 minute. | +| pulsar_function_processed_successfully_1min_total | Counter | The total number of messages processed successfully in the last 1 minute. | | pulsar_function_system_exceptions_total | Counter | The total number of system exceptions. | -| pulsar_function_system_exceptions_total_1min | Counter | The total number of system exceptions in the last 1 minute. | +| pulsar_function_system_exceptions_1min_total | Counter | The total number of system exceptions in the last 1 minute. | | pulsar_function_user_exceptions_total | Counter | The total number of user exceptions. | -| pulsar_function_user_exceptions_total_1min | Counter | The total number of user exceptions in the last 1 minute. | +| pulsar_function_user_exceptions_1min_total | Counter | The total number of user exceptions in the last 1 minute. | | pulsar_function_process_latency_ms | Summary | The process latency in milliseconds. | | pulsar_function_process_latency_ms_1min | Summary | The process latency in milliseconds in the last 1 minute. | | pulsar_function_last_invocation | Gauge | The timestamp of the last invocation of the function. | | pulsar_function_received_total | Counter | The total number of messages received from source. | -| pulsar_function_received_total_1min | Counter | The total number of messages received from source in the last 1 minute. | +| pulsar_function_received_1min_total | Counter | The total number of messages received from source in the last 1 minute. | pulsar_function_user_metric_ | Summary|The user-defined metrics. ## Connectors @@ -617,16 +617,16 @@ Connector metrics contain **source** metrics and **sink** metrics. | Name | Type | Description | |---|---|---| pulsar_source_written_total|Counter|The total number of records written to a Pulsar topic. - pulsar_source_written_total_1min|Counter|The total number of records written to a Pulsar topic in the last 1 minute. + pulsar_source_written_1min_total|Counter|The total number of records written to a Pulsar topic in the last 1 minute. pulsar_source_received_total|Counter|The total number of records received from source. - pulsar_source_received_total_1min|Counter|The total number of records received from source in the last 1 minute. + pulsar_source_received_1min_total|Counter|The total number of records received from source in the last 1 minute. pulsar_source_last_invocation|Gauge|The timestamp of the last invocation of the source. pulsar_source_source_exception|Gauge|The exception from a source. pulsar_source_source_exceptions_total|Counter|The total number of source exceptions. - pulsar_source_source_exceptions_total_1min |Counter|The total number of source exceptions in the last 1 minute. + pulsar_source_source_exceptions_1min_total |Counter|The total number of source exceptions in the last 1 minute. pulsar_source_system_exception|Gauge|The exception from system code. pulsar_source_system_exceptions_total|Counter|The total number of system exceptions. - pulsar_source_system_exceptions_total_1min|Counter|The total number of system exceptions in the last 1 minute. + pulsar_source_system_exceptions_1min_total|Counter|The total number of system exceptions in the last 1 minute. pulsar_source_user_metric_ | Summary|The user-defined metrics. - **Sink** metrics @@ -634,16 +634,16 @@ Connector metrics contain **source** metrics and **sink** metrics. | Name | Type | Description | |---|---|---| pulsar_sink_written_total|Counter| The total number of records processed by a sink. - pulsar_sink_written_total_1min|Counter| The total number of records processed by a sink in the last 1 minute. - pulsar_sink_received_total_1min|Counter| The total number of messages that a sink has received from Pulsar topics in the last 1 minute. + pulsar_sink_written_1min_total|Counter| The total number of records processed by a sink in the last 1 minute. + pulsar_sink_received_1min_total|Counter| The total number of messages that a sink has received from Pulsar topics in the last 1 minute. pulsar_sink_received_total|Counter| The total number of records that a sink has received from Pulsar topics. pulsar_sink_last_invocation|Gauge|The timestamp of the last invocation of the sink. pulsar_sink_sink_exception|Gauge|The exception from a sink. pulsar_sink_sink_exceptions_total|Counter|The total number of sink exceptions. - pulsar_sink_sink_exceptions_total_1min |Counter|The total number of sink exceptions in the last 1 minute. + pulsar_sink_sink_exceptions_1min_total |Counter|The total number of sink exceptions in the last 1 minute. pulsar_sink_system_exception|Gauge|The exception from system code. pulsar_sink_system_exceptions_total|Counter|The total number of system exceptions. - pulsar_sink_system_exceptions_total_1min|Counter|The total number of system exceptions in the last 1 minute. + pulsar_sink_system_exceptions_1min_total|Counter|The total number of system exceptions in the last 1 minute. pulsar_sink_user_metric_ | Summary|The user-defined metrics. ## Proxy