Missing datadog metrics for the prodpublick8s AKS cluster

[dduportal]

Service(s)

Azure, Other

Summary

Datadog dashboards are only reporting a partial set of the metrics from the prodpublick8s cluster

2 Az

ure Kubernetes clusters compared to thes AWS and Digital Ocean clusters as you can see on the screenshots below:

On short term, we have our own Grafana installation to gather metrics for prodpublick8s, but it's running... in this cluster

Reproduction steps

No response

[dduportal]

The logs of the cluster-agent pods for prodpublick8s are filled with the following error message:

2022-09-09 08:04:42 UTC | CLUSTER | INFO | (pkg/clusteragent/admission/controllers/webhook/controller_base.go:171 in processNextWorkItem) | Couldn't reconcile Webhook datadog-webhook: Operation cannot be fulfilled on mutatingwebhookconfigurations.admissionregistration.k8s.io "datadog-webhook": the object has been modified; please apply your changes to the latest version and try again

which looks like DataDog/datadog-agent#10413 and DataDog/datadog-agent#10764

[dduportal]

Closing as it is now working.

[dduportal]

While working on the datadog integration with the artifact caching proxy (#2752 ) we discovered that the datadog agents of this cluster are failing with the following errors:

2022-11-04 17:41:05 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:69 in Error) | check:kubelet | Error running check: [{"message": "Unable to detect the kubelet URL automatically: impossible to reach Kubelet with host: aks-agentpool-34540872-vmss000123. Please check if your setup requires kubelet_tls_verify = false. Activate debug logs to see all attempts made", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1116, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/kubelet/kubelet.py\", line 311, in check\n    raise CheckException(\"Unable to detect the kubelet URL automatically: \" + kubelet_conn_info.get('err', ''))\ndatadog_checks.base.errors.CheckException: Unable to detect the kubelet URL automatically: impossible to reach Kubelet with host: aks-agentpool-34540872-vmss000123. Please check if your setup requires kubelet_tls_verify = false. Activate debug logs to see all attempts made\n"}]

Also a LOT of warnings like below, related to our custom python checks (embedded in the Docker image from jenkins-infra/docker-datadog):

2022-11-04 17:41:05 UTC | CORE | WARN | (pkg/collector/python/datadog_agent.go:125 in LogMessage) | http_check:plugins.jenkins.io:f6eaba3b8709a028 | (http.py:388) | An unverified HTTPS request is being made to https://plugins.jenkins.io/

[dduportal]

Two (closed) issues about the error message that could (or not) help:

• Unable to detect the kubelet URL automatically / cannot validate certificate DataDog/integrations-core#2582
• Unable to detect the kubelet URL automatically after k8s upgrade to 1.16.x DataDog/datadog-agent#5942

[dduportal]

https://github.com/DataDog/helm-charts/blob/a15f3681d724c2cbe406846fefc27c257c50a7e4/examples/datadog/agent_on_aks_values.yaml#L19

Required as of Agent 7.35 because Kubelet certificates in AKS do not have a Subject Alternative Name (SAN) set.

From DataDog/helm-charts#763

Which dates from https://github.com/DataDog/helm-charts/blob/main/charts/datadog/CHANGELOG.md#2320 => https://github.com/jenkins-infra/kubernetes-management/pull/2225/files

[dduportal]

By the way: https://docs.datadoghq.com/agent/troubleshooting/debug_mode/?tab=agentv6v7#containerized-agent is really useful to enable debug log on a given agent while it is running

[dduportal]

Applied and fixed 🥳