Add ability to override cluster_uuid to be used in monitoring data

[ycombinator]

Background and Problem

Logstash monitoring data is stored in .monitoring-logstash-* indices for the Stack Monitoring application. A key field in these documents is cluster_uuid. The value of this field is supposed to be an Elasticsearch cluster UUID. The value determines where in the Stack Monitoring UI the Logstash monitoring data shows up, i.e. which Elasticsearch cluster the data is grouped with.

Starting 7.3.0, the Metricbeat logstash module is able to interrogate various Logstash APIs to collect monitoring data required by Stack Monitoring. Metricbeat builds Elasticsearch documents from this data, intended to be indexed into .monitoring-logstash-* indices.

If a Logstash pipeline is using an Elasticsearch output, that Elasticsearch cluster's UUID is returned in the pipeline vertex object for the output in Logstash API responses. Metricbeat uses this cluster UUID value as the value of the cluster_uuid field in documents it indexes into .monitoring-logstash-*.

However, if a Logstash pipeline is not using an Elasticsearch output, Metricbeat will not index any value for the cluster_uuid field. As a result the Logstash monitoring data will show up under a Standalone Cluster in the Stack Monitoring UI.

A similar thing happens for Beats monitoring data as well, and users are not happy about it. Such users would like to see their monitoring data associated with their production Elasticsearch cluster in the UI.

Knowing how users have responded to Beats monitoring data showing up under Standalone Cluster in the Stack Monitoring UI, wo we'd like to get ahead of this problem for Logstash monitoring data.

Suggested Solution

In the same vein as elastic/beats#13182, add a new optional setting to logstash.yml named xpack.monitoring.cluster_uuid.

Any Logstash APIs that return pipeline representations (vertices and edges) would be affected by this setting. If this setting is set, such APIs should also return a top-level cluster_uuid property.

The logstash Metricbeat module will first look for the top-level cluster_uuid property. If it is present, it will use its value as the value of the cluster_uuid field in .monitoring-logstash-* documents. If its not present, the module will fallback to it's current behavior.

[ycombinator]

Pinging @elastic/stack-monitoring.

[cachedout]

This is mostly done in this PR: #11066

I am blocked on a test issue that I can't get past and I am waiting for the LS to have a little time to look at it. I see that it was just recently rebased, however, so they may be close on this.

[andsel]

@cachedout PR #11106 which exposes cluster_uuid has been merged

[cachedout]

@andsel Thanks! I think it's best to let @ycombinator look over this before we close it and see if there are any outstanding issues to be resolved. Thanks again for your work!

[ycombinator]

Closed via #11106.