Expose file handles count for log and filestream inputs

[lucabelluccini]

Issue

Describe the enhancement:

Monitoring should expose filebeat.harvester.open_files for both log and filestream input

Describe a specific use case for the enhancement or feature:

Ability to track the current open files by Filebeat and eventually expose them in the Stack Monitoring / Dashboards.

Definition of done

• Implement a tracker for the current open files for Log Input
• Implement a tracker for the current open files for Filestream Input
• Expose this tracker
• Tests are in place to validate the tracker contains the right value

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nimarezainia]

this has a high end customer value. The priority should not be adjusted.

[belimawr]

@nimarezainia I started working on this today and realised its partially covered by #35529. It exposes multiple metrics from filestream under the "HTTP" endpoint http://localhost:5066/inputs, those metrics can also be logged in the monitoring logs if logging.metrics.namespaces: ["stats", "dataset"]. The only caveat is the different namespaces and names for the metrics.

I'm wondering if we should unify those metrics and tasks.

[belimawr]

cc: @pierrehilbert.

I'm also changing the status to blocked until we have more clarity on how to proceed to avoid duplicating efforts and having diverging implementations.

[cmacknz]

I suspect that the exact name of the metric here doesn't matter, as long as it is available somewhere. That means the changes in #35529 fundamentally solve the problem of exposing this for filestream.

If we relied entirely on the changes in #35529 though we'd still have a global metric named filebeat.harvester.open_files that will confusingly always read as zero.

Deprecating the global metric or making it conditional on the inputs in use is much more work than just tracking the metric twice under two different names, so I would vote we do that and keep the original implementation plan independent of #35529 .

The metrics counter here is effectively a form of legacy API, and we can't completely abandon an important use case for it because an incompatible v2 API (metrics under /inputs) will exist.

[belimawr]

My understanding is that filebeat.harvester.open_files was created at a time there was only the log input, by what I could gather of the code base so far all those "harvester" metrics are exclusive of the log input, but not "namespaced" as such :/

We could document that the current metrics are only for the log input and document how to get the new ones from filestream (by using the HTTP API or adding the necessary namespace for the logging metrics). What do you think @cmacknz ?

[cmacknz]

Discussed today, I think just documenting that these metrics which could apply to both inputs are only populated by the log input will be confusing.

Outcome was we wait for #35529 to merge, and then duplicate the relevant metrics under filebeat.harvester.open_files.

If we imagine that we eventually add the log input metrics under /inputs, then we can view the global filebeat.harvester.* metrics as the aggregation of the log and filestream metrics where they both mean the same thing. The change above should make these metrics the aggregated view when both filestream and log are running together, we would just be missing support for the log input under /inputs to see only the log input metrics.

[jlind23]

As we are waiting for #35529 I will push this issue out of our current sprint.