OTEL Exporter - multi-tenancy support based on labels

[mdsgrafana]

Is your feature request related to a problem? Please describe.
Tempo supports multitenancy using an HTTP header (such as "X-Scope-OrgID"). With the current implementation of the OTEL exporter, we need to create an exporter per tenant (by setting the header or the endpoint url).

That means each time we want to add a tenant we need to change collector’s configuration or add a new collector, and we should be aware of all tenants of our infrastructure (I want to say exhaustive).

Describe the solution you'd like
A tenant may add (or through an intermediate collector) a label to the metrics representing its name (for instance tenant="123abc"), and the collector may automatically configure the header or the query param accordingly.

Additional context
The Loki exporter recently had this functionality added - open-telemetry/opentelemetry-collector-contrib#3121
And the prometheusremotewriteexporter has this issue for the same function
open-telemetry/opentelemetry-collector-contrib#7945

[TylerHelmuth]

@codeboten @dmitryax @Aneurysm9 @mx-psi @jpkrohling I think this should be transfered to Core but I don't have the permissions to do it.

[jpkrohling]

Done. I think this needs some discussion to figure out what can be done here in a generic fashion. While it would be OK for the Loki exporter to have a native "tenant" attribute which translates to a specific header that is sent to the backend, having the same as part of the OTLP exporter would probably not be appropriate.

I think the cleanest would be to have a client authenticator which would obtain the tenancy data from a static source or resource attributes. What do you think?

[TylerHelmuth]

Gets more complicated if the tenancy needs to come from a dynamic source instead of a piece of telemetry/data from within the collector itself, which feels like a valid situation.

This is an interesting topic, I'll add it to the SIG meeting for 8/17 (the next one I'll be at). I'd like to see a generic solution in the collector to be able to dynamically change things like ID/License/key for exporters/specific exporters.

[newly12]

We also have similar use case, linked PR that introduces headers_setter extension looks very interesting, I am looking forward to its release!

[newly12]

Given the limitation that batch processor discards the context of telemetry data and builds a new one, does it make sense to workaround this by having a pipeline as following, if receiver sets auth info in attributes instead of context,
receiver -> batch processor -> groupbyattrs processor(group by auth attributes) -> a new processor(extract auth attributes and set it in context) -> exporter with headers_setter extension config.

though ideally batch processor should take client info into consideration as mentioned by #4544..

[kovrus]

@jpkrohling open-telemetry/opentelemetry-collector-contrib#12892 should resolve this issue, can you please close it?