[metricbeat] [gcp] add billing additional fields

CHANGELOG.next.asciidoc

@@ -255,7 +255,7 @@ is collected by it.
 - Add GCP Carbon Footprint metricbeat data {pull}34820[34820]
 - Add event loop utilization metric to Kibana module {pull}35020[35020]
 - Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms {pull}36647[36647]
-
+- Enhance GCP billing with detailed tables identification, additional fields, and optimized data handling. {pull}36902[36902]
 
 *Osquerybeat*
 

metricbeat/docs/fields.asciidoc

@@ -35365,6 +35365,60 @@ type: float
 
 --
 
+*`gcp.billing.sku_id`*::
++
+--
+The ID of the resource used by the service.
+
+type: keyword
+
+--
+
+*`gcp.billing.sku_description`*::
++
+--
+A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+
+type: text
+
+--
+
+*`gcp.billing.service_id`*::
++
+--
+The ID of the service that the usage is associated with.
+
+type: keyword
+
+--
+
+*`gcp.billing.service_description`*::
++
+--
+The Google Cloud service that reported the Cloud Billing data.
+
+type: text
+
+--
+
+*`gcp.billing.tags`*::
++
+--
+A collection of key-value pairs that provide additional metadata.
+
+type: object
+
+--
+
+*`gcp.billing.effective_price`*::
++
+--
+The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.
+
+type: float
+
+--
+
 [float]
 === carbon
 

x-pack/metricbeat/module/gcp/billing/_meta/data.json

@@ -16,7 +16,15 @@
             "invoice_month": "202106",
             "project_id": "containerlib-prod-12763",
             "project_name": "elastic-containerlib-prod",
-            "total": 4717.170681
+            "total": 4717.170681,
+            "sku_id": "0D56-2F80-52A5",
+            "service_id": "6F81-5844-456A",
+            "sku_description": "Network Inter Region Ingress from Jakarta to Americas",
+            "service_description": "Compute Engine",
+            "effective_price": 0.00292353,
+            "tags": {
+                "size": "medium"
+            }
         }
     },
     "metricset": {

x-pack/metricbeat/module/gcp/billing/_meta/docs.asciidoc

@@ -1,18 +1,28 @@
 `billing` metricset is designed for collecting billing metrics from Google Cloud
 BigQuery daily cost detail table. BigQuery is a fully-managed, serverless data
 warehouse.
-Cloud Billing export to BigQuery enables you to export detailed Google Cloud
+Cloud Billing export to BigQuery enables you to export standard and detailed Google Cloud
 billing data (such as usage, cost estimates, and pricing data) automatically
 throughout the day to a BigQuery dataset that you specify. Then you can access
 your Cloud Billing data from BigQuery for detailed analysis using Metricbeat.
 Please see https://cloud.google.com/billing/docs/how-to/export-data-bigquery[export
 cloud billing data to BigQuery] for more details on how to export billing data.
 
-In BigQuery dataset, detailed Google Cloud daily cost data is loaded into a data
-table named `gcp_billing_export_v1_<BILLING_ACCOUNT_ID>`. There is a defined
-schema for Google Cloud daily cost data that is exported to BigQuery. Please see
-https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables#data-schema[
-daily cost detail data schema] for more details.
+In BigQuery, Google Cloud daily cost data is categorized into two formats:
+standard and detailed. Each format is stored within a designated dataset and
+follows a structured schema for precise cost analysis. For a comprehensive
+understanding of these formats, consult the https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/standard-usage#standard-usage-cost-data-schema[
+standard] and https://cloud.google.com/billing/docs/how-to/export-data-bigquery-tables/detailed-usage#detailed-usage-cost-data-schema[
+detailed] data schema documentation.
+
+For regular cost monitoring and analysis, standard tables utilize the
+`gcp_billing_export_v1_<BILLING_ACCOUNT_ID>` format, presenting a comprehensive
+view of Google Cloud expenses.
+
+Alternatively, detailed tables, identifiable by the `gcp_billing_export_resource_v1_<BILLING_ACCOUNT_ID>`
+format, offer richer data. They include additional fields, such as `effective_price`,
+enabling a more granular view of expenses.
+
 
 [float]
 === Metricset-specific configuration notes

x-pack/metricbeat/module/gcp/billing/_meta/fields.yml

@@ -15,3 +15,22 @@
   - name: total
     type: float
     description: Total billing amount.
+  - name: sku_id
+    type: keyword
+    description: The ID of the resource used by the service.
+  - name: sku_description
+    type: text
+    description: A description of the resource type used by the service. For example, a resource type for Cloud Storage is Standard Storage US.
+  - name: service_id
+    type: keyword
+    description: The ID of the service that the usage is associated with.
+  - name: service_description
+    type: text
+    description: The Google Cloud service that reported the Cloud Billing data.
+  - name: tags
+    type: object
+    object_type: keyword
+    description: A collection of key-value pairs that provide additional metadata.
+  - name: effective_price
+    type: float
+    description: The charged price for usage of the Google Cloud SKUs and SKU tiers. Reflects contract pricing if applicable, otherwise, it's the list price.