Support ClickHouse server monitoring & Support service hierarchy

[CzyerChen]

Feature #11904

• If this is non-trivial feature, paste the links/URLs to the design doc.
• Update the documentation to include this new feature.
• Tests(including UT, IT, E2E) are added to verify the new feature.
• If it's UI related, attach the screenshots below.

ClickHouse root:

ClickHouse Service:

ClickHouse Instance:

Service Hierarchy:



click CLICKHOUSE node and jump to server monitoring dashboard.

Virtual DB:

• Update the CHANGES log.

[wu-sheng]

The SWIP part should be updated too.

[wu-sheng]

@mrproliu Could you check why ebpf failed?

[wu-sheng]

e2e seems not to be added into GHA control file?

[wu-sheng]

By following my question, the point is, is this process making metrics from two targets in one metric report or two?

[wu-sheng]

@CzyerChen Could you share how the metrics from clickhouse1 and clickhouse2 would be scraped here? One metric reported per target with the same host_name, or one metric for both targets in the same time?

[wankai123]

ClickHouseMetrics_MySQLConnection -> {SampleFamily@26369} 
"SampleFamily(samples=[Sample(name=ClickHouseMetrics_MySQLConnection, labels={net_host_port=9363, job_name=clickhouse-monitoring, 
node_identifier_host_name=clickhouse1, 
http_scheme=http, 
service_instance_id=clickhouse1:9363,
 host_name=clickhouse:8123}, 
value=0.0, timestamp=1709519785903)], context=SampleFamily.RunningContext(metricName=null, meterSamples={}, defaultHistogramBucketUnit=SECONDS))"

The OTEL will scrape and send the metrics node by node, in other words, OAP only received 1 node's metric at one time, so we can't sum the metrics from separate node's metrics.

[CzyerChen]

Have to make it from two targets to two hostnames. Prometheus endpoint fetches data for per node.
I will correct otel-collector-config.yaml.

[CzyerChen]

Mainly sum up labeled value by MQE.

[wu-sheng]

@wankai123 PTAL

[CzyerChen]

Two empty metrics ClickHouseMetrics_KeeperAliveConnections/ClickHouseMetrics_PartMutation answered here: ClickHouse/ClickHouse#60742

meter_clickhouse_instance_connections_alive/meter_clickhouse_connections_alive here will always be 0 because it is from ClickHouse Keeper Prometheus endpoint rather than ClickHouse Prometheus endpoint, and I will remove it.

[wu-sheng]

Are we fetching metrics from ClickHouse Keeper Prometheus endpoint?

[CzyerChen]

ClickHouse Keeper Prometheus endpoint

My Cluster test works with Zookeeper not ClickHouse Keeper. If to test the data from ClickHouse Keeper Prometheus endpoint, we have to try a ClickHouse Keeper cluster.

[wu-sheng]

I see, then, please be clear about this in the docs if you don't intent to build this mode.

[wankai123]

The others LGTM

[wu-sheng]

Please update the SWIP according to the latest implementation and add it to the docs.

[CzyerChen]

• connections_alive means Number of alive connections for embedded ClickHouse Keeper
• add metric of Keeper Outstanding Requets another important metric for embedded ClickHouse Keeper
• update dashboard layout

[CzyerChen]

Please update the SWIP according to the latest implementation and add it to the docs.

ok

[wu-sheng]

SWIP5 needs to attach on SWIP page.

[CzyerChen]

SWIP5 needs to attach on SWIP page.

Ok, linked.

[wu-sheng]

I mean this page should be updated, https://github.com/apache/skywalking/blob/master/docs/en/swip/readme.md. Note the next SWIP number and the following existing SWIPs.

[wu-sheng]

BTW, have you read #11992? Is this helping to polish clickhouse monitoring? Or it is good enough for now.

[CzyerChen]

BTW, have you read #11992? Is this helping to polish clickhouse monitoring? Or it is good enough for now.

It doesn't seem to be necessary on ClickHouse, where the metrics values are all singular values.

The enhancement looks great for multi labels.

[wu-sheng]

Would you like to post dual-language blogs for this new feature?

[CzyerChen]

Would you like to post dual-language blogs for this new feature?

Ok.