[Metricbeat] beat module - add metricset for input metrics

[andrewkroh]

Describe the enhancement:

Add a new input metricset to the Beat module to collect metrics from various inputs with Beats (e.g. aws-s3, winlog, udp). Several of the inputs already export per instance metrics and we are exposing those metrics under /inputs/ on the Beat http monitoring endpoint.

Describe a specific use case for the enhancement or feature:

Once we can collect these metrics we can build dashboards to help understand and diagnose issues with input behavior or configuration. For example this metricset could be enabled as part of Agent monitoring to collect input metrics from Filebeat. Then dashboards could be created in the elastic_agent Fleet integration package for each input type to make sense of the data.

Relates

• [Filebeat] Add /inputs/ route for Filebeat input metrics #33859
• filebeat/input/udp: add initial UDP metrics support #33870
• winlogbeat/eventlog: add initial event processing metrics support #33922

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[botelastic]

This issue doesn't have a Team:<team> label.

[andrewkroh]

On this we are going to pivot to using the http.json metricset with Metricbeat rather than adding new code in the Beat module. This should work given that /inputs/ is returning an array.

So let's build up a Metricbeat configuration and test it standalone. Then we'll prepare to add it into Agent ¹ to monitor Filebeat.

• Agent should only run this when monitoring Filebeat since it's the only Beat with /inputs/ at the current time.
• Add mappings for this data to the metrics-elastic_agent.filebeat data stream.

Footnotes

1. https://github.com/elastic/elastic-agent/blob/ab2cf8a6caea84f48a43e8d61fd4a5ff6aa221bc/internal/pkg/agent/application/monitoring/v1_monitor.go#L514 ↩

[kcreddy]

Thanks for the updates. I will get started and test it standalone

[kcreddy]

CC: @epixa

[andrewkroh]

This was implemented via the http.json metricset in Metricbeat. There was no need to add a new metricset to implement this. See elastic/integrations#5077 and elastic/elastic-agent#2171.