5.7.1 does not work with Kubernetes 1.2

[bakins]

I was able to track it down to the labels the kubelet adds to the containers and are exposed via cadvisor inside the kubelet. in 1.1, we get labels like:

  "io.kubernetes.pod.terminationGracePeriod": "60",
  "io.kubernetes.pod.name": "cluster/kube-datadog-cpwwj"

in 1.2, we get:

"io.kubernetes.container.hash": "d6fe00ff",
  "io.kubernetes.container.name": "POD",
  "io.kubernetes.container.restartCount": "0",
  "io.kubernetes.container.terminationMessagePath": "",
  "io.kubernetes.pod.name": "fluentd-melvb",
  "io.kubernetes.pod.namespace": "cluster",
  "io.kubernetes.pod.terminationGracePeriod": "60",
  "io.kubernetes.pod.uid": "4fec2f14-f2e0-11e5-9e88-42010af20017"

I'm playing with a quick patch, but there's probably a more elegant way.

[bakins]

diff that works in our limited testing:

diff --git a/checks.d/docker_daemon.py b/checks.d/docker_daemon.py
index e6e16c1..447cef7 100644
--- a/checks.d/docker_daemon.py
+++ b/checks.d/docker_daemon.py
@@ -22,6 +22,7 @@ SIZE_REFRESH_RATE = 5  # Collect container sizes every 5 iterations of the check
 MAX_CGROUP_LISTING_RETRIES = 3
 CONTAINER_ID_RE = re.compile('[0-9a-f]{64}')
 POD_NAME_LABEL = "io.kubernetes.pod.name"
+POD_NAMESPACE_LABEL = "io.kubernetes.pod.namespace"

 GAUGE = AgentCheck.gauge
 RATE = AgentCheck.rate
@@ -335,6 +336,9 @@ class DockerDaemon(AgentCheck):
                     if k in labels:
                         v = labels[k]
                         if k == POD_NAME_LABEL and self.is_k8s():
+                            ns = labels.get(POD_NAMESPACE_LABEL)
+                            if ns is not None:
+                                v = "%s/%s" % (ns, v)
                             pod_name = v
                             k = "pod_name"
                             if "-" in pod_name:
diff --git a/checks.d/kubernetes.py b/checks.d/kubernetes.py
index 636109f..9acbe8e 100644
--- a/checks.d/kubernetes.py
+++ b/checks.d/kubernetes.py
@@ -171,10 +171,18 @@ class Kubernetes(AgentCheck):
         tags.append('container_name:%s' % container_name)

         pod_name_set = False
+
+        labels = {}
+
         try:
             for label_name,label in subcontainer['spec']['labels'].iteritems():
-                label_name = label_name.replace('io.kubernetes.pod.name', 'pod_name')
-                if label_name == "pod_name":
+                labels[label_name] = label
+            for label_name,label in labels.iteritems():
+                if label_name == 'io.kubernetes.pod.name':
+                    ns = labels.get('io.kubernetes.pod.namespace')
+                    if ns is not None:
+                        label = "%s/%s" % (ns, label)
+                    label_name = 'pod_name'
                     pod_name_set = True
                     pod_labels = kube_labels.get(label)
                     if pod_labels:

[irabinovitch]

@remh I think you mentioned you were working with @bakins on this one so assigning to you.

[mlamina]

Any news on this one? I'm unable to send any metrics from my Kubernetes cluster to Datadog since I updated the cluster to 1.2.

[bakins]

@mlamina the patch above fixed it for me.

[mlamina]

Oh cool! I am not keen to build the agent myself. Is this patch already integrated and available somewhere on Docker Hub?

[bakins]

I just have a Dockerfile that uses the official as the FROM and patches files in place.

[remh]

@mlamina @bakins working on a patch as well. We'll include it with our next bug fix release

[remh]

@mlamina @bakins
We are actually postponing that to 5.8.0 (which should arrive within a few weeks).
The reason is that we are adding an important new feature that will impact the kubernetes integration.
On top of that, we would like to revisit what tags are being collected, as by default a lot of tags can be collected and not all of them are useful and can end up polluting the graphs editor, etc.