Metricbeat on Kubernetes stores incorrect container.id for pods with multiple containers

[eedugon]

Issue reproduced with Metricbeat 8.4.x and 8.5.x.
Kuberenetes: 1.24

The container metricset from kubernetes module is setting wrong container IDs when pods have multiple containers.

For example:

Both container3 and container4 of the previous example belongs to the same pod, and internally they have different container IDs. But metricbeat for some reason is using the same container.id for all containers of the pod.

Example of POD with multiple containers that can be used to reproduce the issue:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: command-multi-containers-2
  labels:
    purpose: multiple-containers
spec:
  containers:
  - name: container3
    image: endeveit/docker-jq
    env:
    - name: MESSAGE
      value: "field1value|field2value"
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo \"$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")|$MESSAGE container3\"; sleep 10; done"]
  - name: container4
    image: endeveit/docker-jq
    env:
    - name: MESSAGE
      value: "field1value|field2value"
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo \"$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")|$MESSAGE container4\"; sleep 10; done"]
  restartPolicy: OnFailure

This has been reported by some users with Istio enabled (which adds sidecar containers to pods).

Reproducing this is very easy, just configure Metricbeat on Kubernetes with our default manifest suggested here and explore metricbeat data in discover as showed in the screenshot.

[botelastic]

This issue doesn't have a Team:<team> label.