Skip to content

Commit

Permalink
[exporter/awsemfexporter] Add option to disable metric extraction (op…
Browse files Browse the repository at this point in the history 
…en-telemetry#27)
  • Loading branch information
@sky333999
sky333999 authored Jun 2, 2023
1 parent b69f08c commit e88af08
Show file tree
Hide file tree
Showing 6 changed files with 120 additions and 31 deletions.
7 changes: 4 additions & 3 deletions exporter/awsemfexporter/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,13 +32,14 @@ The following exporter configuration parameters are supported.
| `max_retries` | Maximum number of retries before abandoning an attempt to post data. | 1 |
| `dimension_rollup_option` | DimensionRollupOption is the option for metrics dimension rollup. Three options are available: `NoDimensionRollup`, `SingleDimensionRollupOnly` and `ZeroAndSingleDimensionRollup` | "ZeroAndSingleDimensionRollup" (Enable both zero dimension rollup and single dimension rollup) |
| `resource_to_telemetry_conversion` | "resource_to_telemetry_conversion" is the option for converting resource attributes to telemetry attributes. It has only one config onption- `enabled`. For metrics, if `enabled=true`, all the resource attributes will be converted to metric labels by default. See `Resource Attributes to Metric Labels` section below for examples. | `enabled=false` |
| `output_destination` | Specify the EMFExporter output. Currently, two options are available. "cloudwatch" or "stdout" | `cloudwatch` |
| `output_destination` | Specify the EMFExporter output. Currently, two options are available. "cloudwatch" or "stdout" | `cloudwatch` |
| `detailed_metrics` | Retain detailed datapoint values in exported metrics (e.g instead of exporting a quantile as a statistical value, preserve the quantile's population) | `false` |
| `version` | Send metrics to CloudWatchLogs with Embedded Metric Format in selected version [(e.g version 1 with _aws)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html#CloudWatch_Embedded_Metric_Format_Specification_structure), version 0 without _aws) | `1` |
| `disable_metric_extraction` | An option to disable the extraction of metrics from the EMF logs. Setting this to true essentially skips generating and setting the _aws / CloudWatchMetrics section of the EMF log, thus effectively retaining all the fields / labels in the EMF log except for the section responsible for extraction of metrics. | `false` |
| `version` | Send metrics to CloudWatchLogs with Embedded Metric Format in selected version [(e.g version 1 with _aws)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html#CloudWatch_Embedded_Metric_Format_Specification_structure), version 0 without _aws) | `1` |
| `parse_json_encoded_attr_values` | List of attribute keys whose corresponding values are JSON-encoded strings and will be converted to JSON structures in emf logs. For example, the attribute string value "{\\"x\\":5,\\"y\\":6}" will be converted to a json object: ```{"x": 5, "y": 6}``` | [ ] |
| [`metric_declarations`](#metric_declaration) | List of rules for filtering exported metrics and their dimensions. | [ ] |
| [`metric_descriptors`](#metric_descriptor) | List of rules for inserting or updating metric descriptors. | [ ] |
| `retain_initial_value_of_delta_metric` | Specify how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels (e.g. `requests{code=200}`). In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to `true` the first value of all metrics will instead be send to AWS. | false |
| `retain_initial_value_of_delta_metric` | Specify how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels (e.g. `requests{code=200}`). In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to `true` the first value of all metrics will instead be send to AWS. | false |

### metric_declaration
A metric_declaration section characterizes a rule to be used to set dimensions for exported metrics, filtered by the incoming metrics' labels and metric names.
Expand Down
5 changes: 5 additions & 0 deletions exporter/awsemfexporter/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,11 @@ type Config struct {
// Note that at the moment in order to use this feature the value "kubernetes" must also be added to the ParseJSONEncodedAttributeValues array in order to be used
EKSFargateContainerInsightsEnabled bool `mapstructure:"eks_fargate_container_insights_enabled"`

// DisableMetricExtraction is an option to disable the extraction of metrics from the EMF logs.
// Setting this to true essentially skips generating and setting the _aws / CloudWatchMetrics section of the EMF log, thus effectively
// retaining all the fields / labels in the EMF log except for the section responsible for extraction of metrics.
DisableMetricExtraction bool `mapstructure:"disable_metric_extraction"`

// ResourceToTelemetrySettings is an option for converting resource attrihutes to telemetry attributes.
// "Enabled" - A boolean field to enable/disable this option. Default is `false`.
// If enabled, all the resource attributes will be converted to metric labels by default.
Expand Down
22 changes: 22 additions & 0 deletions exporter/awsemfexporter/config_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,28 @@ func TestLoadConfig(t *testing.T) {
logger: zap.NewNop(),
},
},
{
id: component.NewIDWithName(typeStr, "disable_metric_extraction"),
expected: &Config{
AWSSessionSettings: awsutil.AWSSessionSettings{
NumberOfWorkers: 8,
Endpoint: "",
RequestTimeoutSeconds: 30,
MaxRetries: 2,
NoVerifySSL: false,
ProxyAddress: "",
Region: "",
RoleARN: "",
},
LogGroupName: "",
LogStreamName: "",
DimensionRollupOption: "ZeroAndSingleDimensionRollup",
OutputDestination: "cloudwatch",
Version: "1",
DisableMetricExtraction: true,
logger: zap.NewNop(),
},
},
}

for _, tt := range tests {
Expand Down
22 changes: 12 additions & 10 deletions exporter/awsemfexporter/metric_translator.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -162,15 +162,17 @@ func translateGroupedMetricToCWMetric(groupedMetric *groupedMetric, config *Conf
}

var cWMeasurements []cWMeasurement
if len(config.MetricDeclarations) == 0 {
// If there are no metric declarations defined, translate grouped metric
// into the corresponding CW Measurement
cwm := groupedMetricToCWMeasurement(groupedMetric, config)
cWMeasurements = []cWMeasurement{cwm}
} else {
// If metric declarations are defined, filter grouped metric's metrics using
// metric declarations and translate into the corresponding list of CW Measurements
cWMeasurements = groupedMetricToCWMeasurementsWithFilters(groupedMetric, config)
if !config.DisableMetricExtraction { // If metric extraction is disabled, there is no need to compute & set the measurements
if len(config.MetricDeclarations) == 0 {
// If there are no metric declarations defined, translate grouped metric
// into the corresponding CW Measurement
cwm := groupedMetricToCWMeasurement(groupedMetric, config)
cWMeasurements = []cWMeasurement{cwm}
} else {
// If metric declarations are defined, filter grouped metric's metrics using
// metric declarations and translate into the corresponding list of CW Measurements
cWMeasurements = groupedMetricToCWMeasurementsWithFilters(groupedMetric, config)
}
}

return &cWMetrics{
Expand Down Expand Up @@ -376,7 +378,7 @@ func translateCWMetricToEMF(cWMetric *cWMetrics, config *Config) *cwlogs.Event {

// Create EMF metrics if there are measurements
// https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html#CloudWatch_Embedded_Metric_Format_Specification_structure
if len(cWMetric.measurements) > 0 {
if len(cWMetric.measurements) > 0 && !config.DisableMetricExtraction {
if config.Version == "1" {
/* EMF V1
"Version": "1",
Expand Down
Loading

0 comments on commit e88af08

Please sign in to comment.