[8.16](backport #41216) Fix Metricbeat k8s metadata sometimes not being present at startup

[mergify]

Proposed commit message

Fix Metricbeat k8s metadata sometimes not being present at startup

In the metricbeat k8s module, each metricset has an enricher, which is used to add k8s metadata to events. Enrichers use watchers to watch k8s resource changes, and these watchers can be shared between enrichers. This is implemented by each enricher maintaining a writethrough cache of metadata - when a K8s event indicating a change to a resource is received, the metadata cache is updated.

The problem with this is that a watcher is only started once, so enrichers which are created after it was started, have to catch up on the resource change events. This is a great opportunity for race conditions, and is in general unnecessarily complicated.

Instead, we turn the writethrough cache into a readthrough cache. The change event handlers now only invalidate the cache, and the enricher computes the metadata when it is first requested. This completely sidesteps the catch up problem, as we only compute the metadata lazily.

I've also refactored the code a bit:

• Resource event handlers are now added to a watcher when it is created, as opposed to when an enricher is created. They're always the same, so there's no reason to do it multiple times.
• Removed TestBuildMetadataEnricher_EventHandler_PastObjects, as it doesn't seem to check anything not covered by other tests.
• Rearranged the createWatcher function, so actual watcher creation (as opposed to skipping because it's already there) happens on the main path.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Install elastic-agent in K8s using the standalone manifests and track the metadata.

Related issues

• Closes K8s metadata for metricbeat Kubernetes module missing at startup #41213

---

This is an automatic backport of pull request #41216 done by [Mergify](https://mergify.com).

[botelastic]

This pull request doesn't have a Team:<team> label.