Report input level dropped/filtered event metrics

[andrewkroh]

Describe the enhancement:

Today, metrics related to event publishing are available only in an aggregated form that combines metrics from all inputs. This makes it difficult to know which specific input source is responsible for dropped or filtered events.

The input metrics should contain data about the number of events dropped or filtered by each input. This would help narrow down the source of problems to a particular input (and hence a particular integration data stream in the case of Elastic Agent).

I believe this would require the beat publisher client (from libbeat/publisher) to provide a way for inputs to subscribe to this data.

Describe a specific use case for the enhancement or feature:

• Diagnostic dumps from Elastic Agent will contain dropped and filtered metrics that are specific to an integration data stream (via input_metrics.json).
• Input metric dashboards in the Elastic Agent fleet integration can display dropped/filtered charts, providing a visual representation of the issue.
• Standalone beat users can access detailed information about which input is causing drops through the HTTP monitoring API or periodical logs when logging_metrics_namespaces: [stats, inputs] is used. This would allow them to quickly identify and resolve issues.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

We already know the target index/datastream for each event in the output, I wonder if we could instead have per data stream stats at the output level? That might get the same result with less internal pipeline wiring.

[andrewkroh]

I like the idea. However, for standalone Beats users this might not be as useful since most users are writing all data to filebeat-x.x.x. But if it's significantly easier, then perhaps that's a good trade-off.