prometheusremotewriteexporter does not support resource_to_telemetry_conversion

[arjantop-cai]

Is your feature request related to a problem? Please describe.
We would like to distinguish metrics from different deployments based on resource metrics.

Describe the solution you'd like
We use prometheusremotewriteexporter, support for resource_to_telemetry_conversion would be needed.

Describe alternatives you've considered
To my knowledge this is the only way to add more labels/attributes at collector level.

[bogdandrutu]

Describe alternatives you've considered
To my knowledge this is the only way to add more labels/attributes at collector level.

Can you expend on this?

[arjantop-cai]

@bogdandrutu Sorry will expand with our usecase/setup:
We run otel agent on every node and otel collectors.

Agents:

• receive otlp metrics/traces from local pods
• scrape node metrics
• we add a resource label/annotation (node name), mainly to make node metrics per-node

Collector:

• Receives otlp metrics/traces from agents
• Adds resource tag/label (eg. cluster name), to distinguish multiple deployments pushing to the same prometheus instance

So to make node metrics actually per-node and distinguish multiple deployments we would need to convert resource labels into prometheus labels, but to my knowledge this is currently not supported.

Is there another approach that could achieve the same that we are missing? Thanks in advance

[vibhavp]

We've forked opentelemetry-collector to play around with and support for converting opentelemtry resource values to metric labels for prometheusexporter and prometheusremotewriteexporter : https://github.com/edjx/opentelemetry-collector

The functionality can be enabled by adding this block to your exporter's config:

resource_to_telemetry_conversion:
      enabled: true

The fork uses the exporthelper subpackage to implement this functionality, though I have not been able to fully verify it (ResourceToTelemetryConversion seems to be only used by datadogexporter here, so it should be okay). Let me know if this works for you.

[seanhoughton]

I'm also trying to work through this problem with the stackdriver exporter in the opentelemetry-collector-contrib version. I have a resource attribute which I would like to be transformed into a metric label without having to go through the code and set the label on every single usage of the instrument when it's already set up as an attribute.

I think the best place for this type of operation might be in the metric transform processor (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor). This way it would work for all metric exporter types.

[seanhoughton]

I have a working PR that performs this function (linked above). The config looks like this:

processors:
  batch:
  metricstransform:
    transforms:
      - include: "^mymetric_.*"
        match_type: regexp
        action: update
        operations:
          - action: copy_resource_attribute
            attribute_name: "enduser.id"

[odeke-em]

@arjantop-cai @seanhoughton @vibhavp does @krak3n's PR #2899 fix this issue for you?