Metrics-generator: add option to inject the tenant ID as metrics label

[yvrhdn]

Is your feature request related to a problem? Please describe.

Scenario:

• you are running Tempo in multitenant mode
• metrics-generator is enabled but all tenants write to the same Prometheus tenant
• you wish to query metrics for tenants independently

Describe the solution you'd like

Tempo gets a configuration option to add the tenant ID as metric label to all generated metrics.

It would have to be a setting on the registry, something like:

metrics_generator:
  # ...
  registry:
    collection_interval: 15s
    inject_tenant_id: true

Or maybe:

    inject_tenant_id: "tempo_tenant_id"

Describe alternatives you've considered

Workaround: inject an attribute on every span and configure these as dimensions in the service-graph and span-metrics processor. This could be done in the application or by the tracing pipeline.

Downside: doing this in the instrumentation risks missing some spans --> these won't have a tenant ID and won't be counted.

Additional context

Implementation notes: we already inject a label in the registry

tempo/modules/generator/registry/registry.go

Lines 101 to 102 in d94e1f2

	hostname, _ := os.Hostname()
	externalLabels["__metrics_gen_instance"] = hostname

I think we just add some extra code here: if the config setting is active add another externalLabel which contains the value of tenant.
This will automatically add it to all metrics generated for that tenant.

[Jxlam-pdx]

@kvrhdn to resolve escalation#10328 ^^, are there more work aside from #3638?

[yvrhdn]

PR has been merged and will be backported to 2.5.

[Jxlam-pdx]

This is for Ring and they are running GET. The fix is in 2.5 rel being cut this week. The next GET build is 2 weeks out.