fix filter bug in dashboards #10199

ldpliu · 2024-06-06T07:16:46Z

Type of change

Select the type of your PR

Bugfix

kafka dashboard some panels can not filter by Broker.

zookeeper dashboard some pannels can not filter by Node

Description

Please describe your pull request

Checklist

Please go through this checklist and make sure all applicable tasks have been done

Write tests
Make sure all tests pass
Update documentation
Check RBAC rights for Kubernetes / OpenShift roles
Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
Reference relevant issue(s) and close them after merging
Update CHANGELOG.md
Supply screenshots for visual changes, such as Grafana dashboards

scholzj

Thanks for the PR. Any changes to the dashboards have to be done to the packaging/ directory. The examples/ directory is updated only as part of an Strimzi release.

scholzj · 2024-06-06T07:19:13Z

systemtest/src/main/java/io/strimzi/systemtest/utils/specific/MetricsUtils.java

+$ 
+kubectl exec -ti my-cluster-entity-operator-6879c6b9d9-ccldp -c topic-operator -- curl -s localhost:8080/metrics | grep strimzi.resources


This seems wrong?

This seems wrong?

Sorry, removed the change.

fvaleri

@ldpliu I tried the new dashboards and they work fine, but you still have to revert changes in examples/ and helm-charts/ dirs. Just keep changes under packaging/ dir. Thanks.

scholzj · 2024-06-08T20:13:52Z

examples/metrics/grafana-dashboards/strimzi-kafka.json

As I said before, do not change this file.

scholzj · 2024-06-08T20:13:56Z

examples/metrics/grafana-dashboards/strimzi-zookeeper.json

As I said before, do not change this file.

scholzj · 2024-06-08T20:14:13Z

helm-charts/helm3/strimzi-kafka-operator/files/grafana-dashboards/strimzi-kafka.json

As I said before, do not change this file.

scholzj · 2024-06-08T20:14:18Z

helm-charts/helm3/strimzi-kafka-operator/files/grafana-dashboards/strimzi-zookeeper.json

As I said before, do not change this file.

scholzj · 2024-06-08T20:14:33Z

packaging/examples/metrics/grafana-dashboards/strimzi-kafka.json

@@ -1143,7 +1143,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(jvm_memory_used_bytes{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$kafka_broker\",strimzi_io_name=\"$strimzi_cluster_name-kafka\"}) by (kubernetes_pod_name)",
+          "expr": "sum(jvm_memory_bytes_used{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$kafka_broker\",strimzi_io_name=\"$strimzi_cluster_name-kafka\"}) by (kubernetes_pod_name)",


This change needs to be reverted.

scholzj · 2024-06-08T20:14:39Z

packaging/examples/metrics/grafana-dashboards/strimzi-zookeeper.json

@@ -1007,7 +1007,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(jvm_memory_used_bytes{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$zk_node\",strimzi_io_name=\"$strimzi_cluster_name-zookeeper\"}) by (kubernetes_pod_name)",
+          "expr": "sum(jvm_memory_bytes_used{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$zk_node\",strimzi_io_name=\"$strimzi_cluster_name-zookeeper\"}) by (kubernetes_pod_name)",


This change needs to be reverted.

scholzj · 2024-06-08T20:14:47Z

packaging/helm-charts/helm3/strimzi-kafka-operator/files/grafana-dashboards/strimzi-kafka.json

@@ -1143,7 +1143,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(jvm_memory_used_bytes{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$kafka_broker\",strimzi_io_name=\"$strimzi_cluster_name-kafka\"}) by (kubernetes_pod_name)",
+          "expr": "sum(jvm_memory_bytes_used{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$kafka_broker\",strimzi_io_name=\"$strimzi_cluster_name-kafka\"}) by (kubernetes_pod_name)",


This change needs to be reverted.

scholzj · 2024-06-08T20:14:52Z

...ing/helm-charts/helm3/strimzi-kafka-operator/files/grafana-dashboards/strimzi-zookeeper.json

@@ -1007,7 +1007,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(jvm_memory_used_bytes{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$zk_node\",strimzi_io_name=\"$strimzi_cluster_name-zookeeper\"}) by (kubernetes_pod_name)",
+          "expr": "sum(jvm_memory_bytes_used{namespace=\"$kubernetes_namespace\",kubernetes_pod_name=~\"$strimzi_cluster_name-$zk_node\",strimzi_io_name=\"$strimzi_cluster_name-zookeeper\"}) by (kubernetes_pod_name)",


This change needs to be reverted.

scholzj · 2024-06-08T20:16:12Z

packaging/examples/metrics/grafana-dashboards/strimzi-kafka.json

@@ -960,7 +960,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(kubelet_volume_stats_available_bytes{namespace=\"$kubernetes_namespace\",persistentvolumeclaim=~\"data(-[0-9]+)?-$strimzi_cluster_name-(kafka|$pool_name)-[0-9]+\"}) by (persistentvolumeclaim)",
+          "expr": "sum(kubelet_volume_stats_available_bytes{namespace=\"$kubernetes_namespace\",persistentvolumeclaim=~\"data(-[0-9]+)?-$strimzi_cluster_name-$kafka_broker\"}) by (persistentvolumeclaim)",


How does this work with node pools as you are replacing the pool name?

Also, I guess this change should be done to the KRaft dashboard as well?

The $kafka_broker will change based on the pool/node. So the pvc name should be: data(-[0-9]+)?-$strimzi_cluster_name-$kafka_broker
Also changed it in kraft dashboard

But it defaults to something like .* => so that surely doesn't work the same way as what was there. It should be now matching all PVCs which seems more wrong then ignoring the broker selector. Or how does it filter out non-Kafka PVCs?

But it defaults to something like .* => so that surely doesn't work the same way as what was there. It should be now matching all PVCs which seems more wrong then ignoring the broker selector. Or how does it filter out non-Kafka PVCs?

I am not really understand what you mean. The current filter should works. It only filter the pvcs which has name: data(-[0-9]+)?-$strimzi_cluster_name-$kafka_broker

But when no broker is selected, it would pick something like this data(-[0-9]+)?-$strimzi_cluster_name-.* and that should pick for example ZooKeeper volumes in Kafka dashboard and vice versa.

How about this?
kubelet_volume_stats_available_bytes{namespace="multicluster-global-hub",persistentvolumeclaim=~"data(-[0-9]+)?-my-cluster-.*",persistentvolumeclaim!~".*zookeeper(-[0-9]+)?$"}

I'm not sure if that would work in all environments. If yes, it might be an option I guess.

Changed the code to use kubelet_volume_stats_available_bytes{namespace="multicluster-global-hub",persistentvolumeclaim=~"data(-[0-9]+)?-my-cluster-.*",persistentvolumeclaim!~".*zookeeper(-[0-9]+)?$"} please help to review again.

scholzj

@Frawless @im-konge Do you have any environment where you can try it - especially the Available Disk Space with various configurations? I cannot try it right now as I'm traveling until next week.

scholzj · 2024-06-12T07:44:33Z

I guess the pool_name variable is not used anymore and assuming the current code works it can be removed?

ldpliu · 2024-06-12T08:54:53Z

I guess the pool_name variable is not used anymore and assuming the current code works it can be removed?

You are right. Removed

scholzj · 2024-06-15T19:37:58Z

packaging/examples/metrics/grafana-dashboards/strimzi-kraft.json

@@ -329,7 +329,7 @@
      "targets": [
        {
          "datasource": "${DS_PROMETHEUS}",
-          "expr": "sum(kubelet_volume_stats_available_bytes{namespace=\"$kubernetes_namespace\",persistentvolumeclaim=~\"data(-[0-9]+)?-$strimzi_cluster_name-(kafka|$pool_name)-[0-9]+\"}) by (persistentvolumeclaim)",
+          "expr": "sum(kubelet_volume_stats_available_bytes{namespace=\"$kubernetes_namespace\",persistentvolumeclaim=~\"data(-[0-9]+)?-$strimzi_cluster_name-$kafka_broker\", persistentvolumeclaim!~\".*zookeeper(-[0-9]+)?$\"}) by (persistentvolumeclaim)",


I gave this a try. The KRaft dashboard is not using $kafka_broker but $kraft_node. Could you please fix this?

(the same is of course also in the Helm Chart dashboard)

I gave this a try. The KRaft dashboard is not using $kafka_broker but $kraft_node. Could you please fix this?

Changed, please help to try again

Signed-off-by: ldpliu <[email protected]>

Frawless · 2024-06-17T11:06:22Z

@Frawless @im-konge Do you have any environment where you can try it - especially the Available Disk Space with various configurations? I cannot try it right now as I'm traveling until next week.

hey, sorry for delay, I just tried the changes and all of them works fine for my deployment. I used latest changes that were committed today.

scholzj · 2024-06-17T11:39:28Z

Thanks for the PR @ldpliu

scholzj requested changes Jun 6, 2024

View reviewed changes

ldpliu force-pushed the fix-dashboard branch 2 times, most recently from 8f2a0af to d9f4c11 Compare June 6, 2024 23:47

ldpliu closed this Jun 7, 2024

ldpliu force-pushed the fix-dashboard branch from d9f4c11 to e9d980a Compare June 7, 2024 00:09

ldpliu reopened this Jun 7, 2024

ldpliu force-pushed the fix-dashboard branch from 0496444 to 9dc3b51 Compare June 7, 2024 00:24

fvaleri self-requested a review June 8, 2024 13:50

fvaleri reviewed Jun 8, 2024

View reviewed changes

scholzj reviewed Jun 8, 2024

View reviewed changes

ldpliu force-pushed the fix-dashboard branch from 9dc3b51 to 41ce42e Compare June 11, 2024 08:55

scholzj added this to the 0.42.0 milestone Jun 11, 2024

ldpliu force-pushed the fix-dashboard branch 2 times, most recently from 48c86dd to daf94c9 Compare June 12, 2024 06:04

scholzj reviewed Jun 12, 2024

View reviewed changes

ldpliu force-pushed the fix-dashboard branch from daf94c9 to 1a5e5f6 Compare June 12, 2024 08:42

scholzj requested changes Jun 15, 2024

View reviewed changes

fix filter bug in dashboards

de30656

Signed-off-by: ldpliu <[email protected]>

ldpliu force-pushed the fix-dashboard branch from 1a5e5f6 to de30656 Compare June 17, 2024 00:29

Frawless mentioned this pull request Jun 17, 2024

Fix filter for Kafka's and ZK's storage in dashboards skodjob/deployment-hub#169

Merged

Frawless approved these changes Jun 17, 2024

View reviewed changes

scholzj approved these changes Jun 17, 2024

View reviewed changes

ppatierno approved these changes Jun 17, 2024

View reviewed changes

scholzj merged commit 654cd6d into strimzi:main Jun 17, 2024
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix filter bug in dashboards #10199

fix filter bug in dashboards #10199

ldpliu commented Jun 6, 2024 •

edited

Loading

scholzj left a comment

scholzj Jun 6, 2024

ldpliu Jun 6, 2024

fvaleri left a comment

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

scholzj Jun 8, 2024

ldpliu Jun 11, 2024 •

edited

Loading

scholzj Jun 11, 2024

ldpliu Jun 11, 2024

scholzj Jun 11, 2024

ldpliu Jun 11, 2024

scholzj Jun 11, 2024

ldpliu Jun 12, 2024

scholzj left a comment

scholzj commented Jun 12, 2024

ldpliu commented Jun 12, 2024

scholzj Jun 15, 2024

scholzj Jun 15, 2024

ldpliu Jun 17, 2024

Frawless commented Jun 17, 2024

scholzj commented Jun 17, 2024

		$
		kubectl exec -ti my-cluster-entity-operator-6879c6b9d9-ccldp -c topic-operator -- curl -s localhost:8080/metrics \| grep strimzi.resources

fix filter bug in dashboards #10199

fix filter bug in dashboards #10199

Conversation

ldpliu commented Jun 6, 2024 • edited Loading

Type of change

Description

Checklist

scholzj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fvaleri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldpliu Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scholzj left a comment

Choose a reason for hiding this comment

scholzj commented Jun 12, 2024

ldpliu commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Frawless commented Jun 17, 2024

scholzj commented Jun 17, 2024

ldpliu commented Jun 6, 2024 •

edited

Loading

ldpliu Jun 11, 2024 •

edited

Loading