Cherry-pick #18172 to 7.x: [Metricbeat] Add dashboard for googlecloud storage metricset

CHANGELOG.next.asciidoc

@@ -448,6 +448,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Move the perfmon metricset to GA. {issue}16608[16608] {pull}17879[17879]
 - Stack Monitoring modules now auto-configure required metricsets when `xpack.enabled: true` is set. {issue}16471[[16471] {pull}17609[17609]
 - Add static mapping for metricsets under aws module. {pull}17614[17614] {pull}17650[17650]
+- Add dashboard for googlecloud storage metricset. {pull}18172[18172]
 - Collect new `bulk` indexing metrics from Elasticsearch when `xpack.enabled:true` is set. {issue} {pull}17992[17992]
 - Remove requirement to connect as sysdba in Oracle module {issue}15846[15846] {pull}18182[18182]
 - Update MSSQL module to fix some SSPI authentication and add brackets to USE statements {pull}17862[17862]]

metricbeat/docs/modules/googlecloud.asciidoc

@@ -144,14 +144,14 @@ For example, if Compute Metricset fetches 14 metrics (which is the number of met
 
 [float]
 == Metricsets
-Currently, we have `compute`, `loadbalancing`, and `pubsub` metricset in
-`googlecloud` module.
+Currently, we have `compute`, `loadbalancing`, `pubsub`, `stackdriver` and
+`storage` metricset in `googlecloud` module.
 
 [float]
 === `compute`
-Compute Metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine]
-Virtual Machines in Google Cloud Platform. The `compute` metricset contains some of the
-metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API].
+This metricset fetches metrics from https://cloud.google.com/compute/[Compute Engine]
+Virtual Machines in Google Cloud Platform. The `compute` metricset contains some
+of the metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[GCP Compute Monitoring API].
 Extra labels and metadata are also extracted using the https://cloud.google.com/compute/docs/reference/rest/v1/instances/get[Compute API].
 This is enough to get most of the info associated with a metric like compute
 labels and metadata and metric specific Labels.
@@ -160,6 +160,45 @@ The `compute` metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-googlecloud-compute-overview.png[]
 
+[float]
+=== `loadbalancing`
+This metricset fetches metrics from https://cloud.google.com/load-balancing/[Load Balancing]
+in Google Cloud Platform. The `loadbalancing` metricset contains all metrics
+exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing[GCP Load Balancing Monitoring API].
+
+[float]
+=== `pubsub`
+This metricset fetches metrics from https://cloud.google.com/pubsub/[Pub/Sub]
+topics and subscriptions in Google Cloud Platform. The `pubsub` metricset
+contains all GA stage metrics exported from the
+https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[GCP PubSub Monitoring API].
+
+The `pubsub` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-pubsub-overview.png[]
+
+[float]
+=== `stackdriver`
+Stackdriver provides visibility into the performance, uptime, and overall health
+of cloud-powered applications. It collects metrics, events, and metadata from
+different services from Google Cloud.
+This metricset is to collect https://cloud.google.com/monitoring/api/metrics_gcp[monitoring metrics]
+from Google Cloud using `ListTimeSeries` API.
+
+[float]
+=== `storage`
+This metricset fetches metrics from https://cloud.google.com/storage/[Storage]
+in Google Cloud Platform. The `storage` metricset contains all GA metrics
+exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-storage[GCP Storage Monitoring API].
+
+We recommend users to define `period: 5m` for this metricset because in Google
+Cloud, storage monitoring metrics are written every 5-minute sample period with
+a 10-minute ingest delay.
+
+The `storage` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-storage-overview.png[]
+
 
 [float]
 === Example configuration
@@ -173,11 +212,11 @@ metricbeat.modules:
 - module: googlecloud
   metricsets:
     - compute
-  region: "us-central1"
+  region: "us-"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 300s
+  period: 1m
 
 - module: googlecloud
   metricsets:
@@ -188,25 +227,15 @@ metricbeat.modules:
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 60s
+  period: 1m
 
 - module: googlecloud
   metricsets:
     - storage
-  region: "us-central1"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 300s
-
-- module: googlecloud
-  metricsets:
-    - compute
-  region: "us-"
-  project_id: "your project id"
-  credentials_file_path: "your JSON credentials file path"
-  exclude_labels: false
-  period: 60s
+  period: 5m
 ----
 
 [float]

x-pack/metricbeat/metricbeat.reference.yml

@@ -498,11 +498,11 @@ metricbeat.modules:
 - module: googlecloud
   metricsets:
     - compute
-  region: "us-central1"
+  region: "us-"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 300s
+  period: 1m
 
 - module: googlecloud
   metricsets:
@@ -513,25 +513,15 @@ metricbeat.modules:
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 60s
+  period: 1m
 
 - module: googlecloud
   metricsets:
     - storage
-  region: "us-central1"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 300s
-
-- module: googlecloud
-  metricsets:
-    - compute
-  region: "us-"
-  project_id: "your project id"
-  credentials_file_path: "your JSON credentials file path"
-  exclude_labels: false
-  period: 60s
+  period: 5m
 
 #------------------------------- Graphite Module -------------------------------
 - module: graphite

x-pack/metricbeat/module/googlecloud/_meta/config.yml

@@ -1,11 +1,11 @@
 - module: googlecloud
   metricsets:
     - compute
-  region: "us-central1"
+  region: "us-"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 300s
+  period: 1m
 
 - module: googlecloud
   metricsets:
@@ -16,22 +16,12 @@
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 60s
+  period: 1m
 
 - module: googlecloud
   metricsets:
     - storage
-  region: "us-central1"
-  project_id: "your project id"
-  credentials_file_path: "your JSON credentials file path"
-  exclude_labels: false
-  period: 300s
-
-- module: googlecloud
-  metricsets:
-    - compute
-  region: "us-"
   project_id: "your project id"
   credentials_file_path: "your JSON credentials file path"
   exclude_labels: false
-  period: 60s
+  period: 5m

x-pack/metricbeat/module/googlecloud/_meta/docs.asciidoc

@@ -134,18 +134,57 @@ For example, if Compute Metricset fetches 14 metrics (which is the number of met
 
 [float]
 == Metricsets
-Currently, we have `compute`, `loadbalancing`, and `pubsub` metricset in
-`googlecloud` module.
+Currently, we have `compute`, `loadbalancing`, `pubsub`, `stackdriver` and
+`storage` metricset in `googlecloud` module.
 
 [float]
 === `compute`
-Compute Metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine]
-Virtual Machines in Google Cloud Platform. The `compute` metricset contains some of the
-metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API].
+This metricset fetches metrics from https://cloud.google.com/compute/[Compute Engine]
+Virtual Machines in Google Cloud Platform. The `compute` metricset contains some
+of the metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[GCP Compute Monitoring API].
 Extra labels and metadata are also extracted using the https://cloud.google.com/compute/docs/reference/rest/v1/instances/get[Compute API].
 This is enough to get most of the info associated with a metric like compute
 labels and metadata and metric specific Labels.
 
 The `compute` metricset comes with a predefined dashboard:
 
 image::./images/metricbeat-googlecloud-compute-overview.png[]
+
+[float]
+=== `loadbalancing`
+This metricset fetches metrics from https://cloud.google.com/load-balancing/[Load Balancing]
+in Google Cloud Platform. The `loadbalancing` metricset contains all metrics
+exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing[GCP Load Balancing Monitoring API].
+
+[float]
+=== `pubsub`
+This metricset fetches metrics from https://cloud.google.com/pubsub/[Pub/Sub]
+topics and subscriptions in Google Cloud Platform. The `pubsub` metricset
+contains all GA stage metrics exported from the
+https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[GCP PubSub Monitoring API].
+
+The `pubsub` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-pubsub-overview.png[]
+
+[float]
+=== `stackdriver`
+Stackdriver provides visibility into the performance, uptime, and overall health
+of cloud-powered applications. It collects metrics, events, and metadata from
+different services from Google Cloud.
+This metricset is to collect https://cloud.google.com/monitoring/api/metrics_gcp[monitoring metrics]
+from Google Cloud using `ListTimeSeries` API.
+
+[float]
+=== `storage`
+This metricset fetches metrics from https://cloud.google.com/storage/[Storage]
+in Google Cloud Platform. The `storage` metricset contains all GA metrics
+exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-storage[GCP Storage Monitoring API].
+
+We recommend users to define `period: 5m` for this metricset because in Google
+Cloud, storage monitoring metrics are written every 5-minute sample period with
+a 10-minute ingest delay.
+
+The `storage` metricset comes with a predefined dashboard:
+
+image::./images/metricbeat-googlecloud-storage-overview.png[]