[Metricbeat] Enhancement for stackdriver metricset

[kaiyan-sheng]

This issue is to track enhancements for googlecloud stackdriver metricset:

• Add google doc link to the documentation, so that it’s easier to find what GCP metrics are available ([Metricbeat] Update googlecloud stackdriver metricset documentation #18693)

• Research if there is a separate API call we can make to get a list of metric types(like the list shown in google doc link above), then we can implement the wildcard into stackdriver metricset (Accept prefix as metric_types for stackdriver metricset in GCP #19345)

• Remove prefix from metric_type in googlecloud config. For example:
  compute.googleapis.com/instance/cpu/utilization -> instance/cpu/utilization (Move service config under stackdriver.metrics and simplify metric_types #18691)

• Use service parameter in config to construct the full metric_type. compute.googleapis.com/instance/cpu/utilization = compute + .googleapis.com/ + metric_type (Move service config under stackdriver.metrics and simplify metric_types #18691)

• Improve documentation around ALIGNER https://www.elastic.co/guide/en/beats/metricbeat/7.8/metricbeat-metricset-googlecloud-stackdriver.html#_metricset_config_and_parameters ([Metricbeat] Update googlecloud stackdriver metricset documentation #18693)

• Rename stackdriver metricset ([Metricbeat] Enhancement for stackdriver metricset #18664)

cc @sorantis Please feel free to add more items. Thank you for the feedback!

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[sorantis]

something to consider: should the service parameter accept just a service name or full service prefix, i.e. compute.googleapis.com/ vs. compute

[kaiyan-sheng]

Do you know if google cloud metric type will have other prefix than googleapis.com? I also checked the custom monitoring metric names, which are also begin with custom.googleapis.com. If the googleapis.com part is always common among all monitoring metrics, then I think it's better to keep service config parameter to be compute.

[sorantis]

Do you know if google cloud metric type will have other prefix than googleapis.com? I also checked the custom monitoring metric names, which are also begin with custom.googleapis.com. If the googleapis.com part is always common among all monitoring metrics, then I think it's better to keep service config parameter to be compute.

Agreed. In that case it makes sense to keep just the service names.

A larger question. Is stackdriver actually the right name for the metricset? Google is moving away from Stackdriver to Operations (see here). Maybe we should align it before it's gained large adoption.

[kaiyan-sheng]

Regarding to the naming, more specifically, we are using monitoring under Operations/stackdriver in Metricbeat. I thought about renaming stackdriver metricset to monitoring metricset (similar to our azure monitor metricset). But my guess is stackdriver is still the most popular naming convention in users? @sayden What do you think?

[sayden]

I think it's a very good idea. A user will look for a "monitoring solution for google cloud" but not a "stackdriver module" or a "stackdriver monitoring module"

[sayden]

BTW, regarding the list of metrics, I think I read about it here: https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors/list

[sorantis]

+1 I think we should align with the official naming used by google cloud. We can add it as another item to the original list.

[kaiyan-sheng]

@sayden Yes, we are already using metricDescriptors already in the code so it would be leveraging the same API response body or similar. Thanks!!

[kaiyan-sheng]

@sorantis @sayden I'm very bad at naming 😂 I suggest we rename stackdriver metricset to monitoring metricset. WDYT? Or operation metricset?

[kaiyan-sheng]

More thinking about renaming stackdriver metricset:

1. Operations: this is the official name to replace stackdriver. But it does not seem to be used in GCP portal yet.
   Enter operations - compute engine showed up but no monitoring

Enter stackdriver - logging, monitoring, debugger, error reporting and profiler showed up

Enter monitoring - actual monitoring service showed up

2. Google Cloud Metrics Documentation is under Operation Suite -> Monitoring:

3. One thing bad about renaming this metricset to monitoring:

`monitoring` is also a service name under Google Cloud Metrics. We would have a conflict of metricset name if we decide to add this into Metricbeat metricset using light module.

With all above, WDYT about renaming stackdriver to metrics? @sorantis @sayden @exekias

[exekias]

metrics sounds like a good option to me, having that we may end up wanting a monitoring metricset to cover monitoring service metrics

[andresrc]

it seems the second item is also done, can we close this issue?

[kaiyan-sheng]

@andresrc Yes everything is done for this issue. Thanks for checking!