New Agent pools do not have kubelet metrics

[Aaron-ML]

What happened:
Added secondary node pools due to the inflexibilities of the original nodepool.

What you expected to happen:
Expect non-default Nodepools to have kubelet stats to allow for container stat monitoring.

How to reproduce it (as minimally and precisely as possible):

Add new nodepool on existing AKS cluster, setup prometheus to scrape kubelet http-metrics from each node. Observe only the default nodepool is reachable via http://<node_IP>:10255/metrics

Anything else we need to know?:

We currently use kublet to provide container level metrics from prometheus such as CPU/memory stats among other things.

Support Ticket: 120050624005838

Environment:

• Kubernetes version (use kubectl version):

v1.16.7

• Size of cluster (how many worker nodes are in the cluster?)

This is happening on our smaller dev AKS clusters, stage and production currently. So anywhere from 1-15 nodes that vary in VM size.

• General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)

Webservices/Java JVM applications

• Others:
  This is a huge pain point for us as we can't troubleshoot any container resources in our new primary nodepools.

[andyzhangx]

port 10255 is HTTP, it's not secure, pls switch to use 10250, here is an example about how to set config in prometheus to use port 10250
https://github.com/prometheus/prometheus/blob/5bb7f00d00ba2d73488630851b352974511c233a/documentation/examples/prometheus-kubernetes.yml#L65-L73

[embik]

@andyzhangx has the plain http port on 10255 been deactivated without mention in the changelog? This worked on clusters deployed a few months ago, so very surprised to see it gone.

[andyzhangx]

cc @palma21

[Aaron-ML]

@andyzhangx I've attempted to set it to https:10250 but I get "Unauthorized" as a response.

Note that this is still working on the original node pool... so I'm not sure what's actually changed on the new nodepool. Can you clarify?

[Aaron-ML]

@andyzhangx @palma21

Can we open this ticket back up? Not sure that anything got resolved here.

I've attempted to scale up the original node pool and can confirm those new nodes have working kubelet endpoints.

I've also scaled up the new nodepool and those new nodes do not have working endpoints for kubelet

[lBowlin]

@andyzhangx We attempted to use https:10250 and were not successful. Can you suggest other steps to try? Or why we are getting "Unauthorized" when trying to use this port?

We only see this issue on secondary nodepools. It does not occur on the primary nodepool.

[palma21]

Mentioned on release notes here: https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2020-01-27

Did you upgrade your original nodepool to 1.16 or was it created on 1.16?

there is currently a bug that is being fixed where the upgrade did not pick up that change, so you might have 12255 working on that one. New pools created originally on 1.16 would not. That could explain your different pool behavior, could you confirm?

Kubelet port 10255 is disabled by default:
kubernetes/kubernetes#59666 (comment)
it is required to either use healthz port:
kubernetes/kubernetes#63812
or 10250 with required authorization

I believe in your case what is missing is needed flags on kubelet which we are enabling right now as well. In the meantime you can workaround it with and confirm if that solves it?
https://github.com/jnoller/kubernaughty/blob/master/tools/enable-webhook

[Aaron-ML]

Mentioned on release notes here: https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2020-01-27

Did you upgrade your original nodepool to 1.16 or was it created on 1.16?

there is currently a bug that is being fixed where the upgrade did not pick up that change, so you might have 12255 working on that one. New pools created originally on 1.16 would not. That could explain your different pool behavior, could you confirm?

Kubelet port 10255 is disabled by default:
kubernetes/kubernetes#59666 (comment)
it is required to either use healthz port:
kubernetes/kubernetes#63812
or 10250 with required authorization

I believe in your case what is missing is needed flags on kubelet which we are enabling right now as well. In the meantime you can workaround it with and confirm if that solves it?
https://github.com/jnoller/kubernaughty/blob/master/tools/enable-webhook

Thanks for responding and reopening the ticket!

Looks like it was referenced under "Azure Monitor" on release notes so I missed it, that's my mistake.

The original node pool was created on an older version but was upgraded to 1.16 including the control plane, the secondary nodepool came after. The issue we were concerned with was the difference between the two, the bug you mention sounds like it could be what we are seeing.

We've since setup Prometheus authorization over https to scrape the new endpoint. Do you know if this bug is isolated to this endpoint or are there other things we should be looking at as well?

[ShaggO]

port 10255 is HTTP, it's not secure, pls switch to use 10250, here is an example about how to set config in prometheus to use port 10250
https://github.com/prometheus/prometheus/blob/5bb7f00d00ba2d73488630851b352974511c233a/documentation/examples/prometheus-kubernetes.yml#L65-L73

System information/context:
I've just upgraded an "old" cluster (1.14.6 -> 1.15.11 -> 1.16.9) that uses availability sets and now the http endpoint at port 10255 is missing. I've switched to port 10250 and set up a service account bound to a clusterrole with appropriate rules:

- apiGroups: [""]
  resources: ["nodes/stats", "nodes/metrics"]
  verbs: ["get"]

I can now query https://${NODE_IP}:10250/metrics and https://${NODE_IP}:10250/stats/summary with the bearer token header from my serviceaccount but I don't have the CA bundle to verify the connection/certificate.

Problem: I want to verify the "self-signed" kubelet certificate, but the serviceaccount ca.crt does not work. Said ca.crt works for verifying the master node api endpoint but not the nodes' kubelet endpoint.
Is there a way to verify the nodes' kubelet certificate(s)/get the nodes' kubelet CA bundle(s) making me able to verify the connection?

[StianOvrevage]

For those of you using ServiceMonitor CRDs this worked for me.

Update the prometheus-operator-kubelet ServiceMonitor, in our case it's in the monitoring namespace.

kubectl edit servicemonitor -n monitoring prometheus-operator-kubelet

Change from http to https where applicable, and add tlsConfig.insecureSkipVerify: true.

  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 10s
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    interval: 10s
    path: /metrics/cadvisor
    port: https-metrics
    scheme: https
    tlsConfig:
      insecureSkipVerify: true

Now both kubelet+cadvisor targets in Prometheus are up and metrics are once again flowing :)