Pod and container resource limit metrics missing intermittently

[swiatekm]

In #41216, calculating metadata for events in the metricbeat kubernetes module was made eager. As a result, it is only calculated when an event needs it, cached, and the cache is invalidated when an update arrives from the API Server. However, the update code not only computes metadata, but also calculates the limit metrics. This now only happens when we attach Pod metadata to an event for the first time after an update, so the presence of these metrics depends on the ordering of metric fetches.

In the short term, we should decouple these metrics updates from metadata enrichment and have the watcher apply them directly. In the longer term, maybe they can be made eager as well?

For confirmed bugs, please report:

• Version: 8.15.4-SNAPSHOT, 8.16.0-SNAPSHOT, 8.17.0-SNAPSHOT, 9.0.0-SNAPSHOT
• Steps to Reproduce:
  • Start metricbeat in K8s with the kubernetes module enabled
  • Wait until one of the Pods is updated
  • The limit metrics should be missing

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[nkvoll]

Since this is marked as "intermittently": We've described what can cause it to become missing -- what causes these metrics to become available again?

[swiatekm]

Since this is marked as "intermittently": We've described what can cause it to become missing -- what causes these metrics to become available again?

They come back if other metrics related to Pods are fetched first, and there's no updates to Pods on the Node in the meantime. So, for example:

1. We fetch Pod state metrics
2. We'd compute metadata for them, also computing the resource limit metrics as a side effect
3. We fetch Pod kubelet metrics (no Pod update happens between here and point 2)