[8.x](backport #42398) Handle leak of process info in hostfs provider for add_session_metadata

[mergify]

Proposed commit message

Fixes #42317

So, it turns out that the processsDB used by the procfs provider in add_session_metadata expects events to come in order, which won't always be the case under load. If we get a an exit event before the exec event, we'll drop the exit event, and then the process event will remain in the db.processes map indefinitely. In addition to this, auditbeat is configured to tell netlink to drop events, meaning that under load, we can lose either the exec or the exit event, potentially leading to a leak if we can never pair up the two for a given process.

This alters the DB so we don't drop orphaned exit events, and instead the DB reaper will wait a few iterations of reapProcs() to try to match the orphaned exit. We also optionally reap process exec events. I've tested this under load, and it does prevent the process DB from growing indefinitely.

There's a few caveats to this as-is:

• We're now putting every single exit event into our db.removalMap, which means we'll be using more memory until those exit events are reaped. I can't really think of a good way around this.
• This processor still uses a lot of resources, and under high-load situations, we may still end up using an unacceptable amount of memory.
• If we need to reap processes, it can result in data loss if the processes don't exist in /proc.

There's also a few smaller changes to the process DB:

• The removal list has been changed from a heap type to a map. This is less performant, but needed, as we're looking up exit events with every exec.
• We expose a number of new config vars.
• This adds metrics to the DB, to further help out with any issues in the future.

I'm still running performance tests on this, as the behavior is a bit bursty and hard to measure without some proper scripts. Will update when I have results.

How to test

Run auditbeat with the following:

- module: auditd
  # Load audit rules from separate files. Same format as audit.rules(7).
  audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
  audit_rules: |
    -a exit,always -F arch=b64 -S fork
    -a exit,always -F arch=b64 -S vfork
    ## set_sid
    -a exit,always -F arch=b64 -F euid=0 -S execve -k rootact
    -a exit,always -F arch=b32 -F euid=0 -S execve -k rootact
    -a always,exit -F arch=b64 -S connect -F a2=16 -F success=1 -F key=network_connect_4
    -a always,exit -F arch=b64 -F exe=/bin/bash -F success=1 -S connect -k "remote_shell"
    -a always,exit -F arch=b64 -F exe=/usr/bin/bash -F success=1 -S connect -k "remote_shell" 
    -a always,exit -F arch=b64 -S exit_group
    -a exit,always -F arch=b64 -S close
    -a always,exit -F arch=b64 -S exit
    -a exit,always -F arch=b64 -S kill
    -a always,exit -F arch=b64 -S setsid 
    -a always,exit -F arch=b64 -S execve,execveat -k exec

processors:
  - add_session_metadata:
      backend: "procfs"

logging.level: debug

Grep for the REAPER: log line to examine the following the state of the various DB maps.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Data: ProcessDB under load

Data: Memory during and after load

---

This is an automatic backport of pull request #42398 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)