Remove unbounded cardinality internal metric labels

[jszwedko]

Some Vector components publish internal metrics with unbounded tag cardinality. These include:

• file source is tagging metrics with a dynamic file tag
• kubernetes_logs is tagging metrics with a dynamic file tag
• TODO: identify other sources

This was improved in v0.24.0 with the addition of the ability to expire these high cardinality metrics, but I still think we should go further and remove these labels (or let users opt into them) to avoid unexpected cardinality issues in Vector or downstream systems that the internal telemetry is being sent to.

Ref:

• Excessive component_received_bytes_total internal metrics since 0.21 #13408
• Throttle config option to disable "key" label in events_discarded_total metric #17778

[pachico]

I can subscribe to the users affected by this problem.
Internal metrics cardinality is much bigger (x10) than the rest of our metrics, and we run hundreds of thousands of active series.

[dmuth]

Subscribing--I wonder if this might be related to #16895 which I filed a few weeks ago--I can make Vector slow down in weird ways if I have a single sink writing to 10,000+ files.

[eposinitskiy]

We are facing this issue with fluent source, metrics are emitted with tag peer_addr containing port.
In our case, vector aggregator is situated behind the AWS NLB and we see peer_addr always containing different port, apparently NLB doesn't keep connection open. On the fluentd side we have double checked the keepalive, and don't see connection being re-opened frequently.

This eventually results into 300K+ unique TS metrics on the vector side.

[dsmith3197]

Note that peer_addr has been removed in the v0.34.0 release (#15426).

[dsmith3197]

the file tag has also been made opt-in #19084.

the key tag for the throttle transform has been made opt-in #19083.

[dsmith3197]

From our analysis, the above covers all of the internal metric labels with unbounded cardinality.

Please report here if you find others.

[jszwedko]

Reopening this since we found another potential one: #19447 (comment)