Unable to detect the kubelet URL automatically

[hden]

Output of the info page

Getting the status from the agent.

===============
Agent (v6.10.2)
===============

  Status date: 2019-04-11 13:04:51.894541 UTC
  Pid: 376
  Python Version: 2.7.15
  Logs: 
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -399µs
    System UTC time: 2019-04-11 13:04:51.894541 UTC

  Host Info
  =========
    bootTime: 2019-04-09 01:35:59.000000 UTC
    kernelVersion: 4.14.104-95.84.amzn2.x86_64
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: buster/sid
    procs: 71
    uptime: 59h23m57s

  Hostnames
  =========
    ec2-hostname: ip-192-168-95-176.ap-northeast-1.compute.internal
    hostname: i-0edb28b7904a45eb3
    instance-id: i-0edb28b7904a45eb3
    socket-fqdn: datadog-agent-4dfn2
    socket-hostname: datadog-agent-4dfn2
    hostname provider: aws
    unused hostname providers:
      configuration/environment: hostname is empty
      gce: unable to retrieve hostname from GCE: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname

=========
Collector
=========

  Running Checks
  ==============
    
    cpu
    ---
      Instance ID: cpu [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 6, Total: 114
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    disk (2.1.0)
    ------------
      Instance ID: disk:e5dffb8bef24336f [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 190, Total: 3,800
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 93ms
      
    
    docker
    ------
      Instance ID: docker [�[32mOK�[0m]
      Total Runs: 19
      Metric Samples: Last Run: 48, Total: 912
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 19
      Average Execution Time : 37ms
      
    
    file_handle
    -----------
      Instance ID: file_handle [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 5, Total: 100
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    io
    --
      Instance ID: io [�[32mOK�[0m]
      Total Runs: 19
      Metric Samples: Last Run: 39, Total: 714
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    kubelet (2.4.0)
    ---------------
      Instance ID: kubelet:d884b5186b651429 [�[31mERROR�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 6ms
      Error: Unable to detect the kubelet URL automatically.
      Traceback (most recent call last):
        File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py", line 774, in run
          self.check(copy.deepcopy(self.instances[0]))
        File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/kubelet/kubelet.py", line 124, in check
          raise CheckException("Unable to detect the kubelet URL automatically.")
      CheckException: Unable to detect the kubelet URL automatically.
    
    kubernetes_apiserver
    --------------------
      Instance ID: kubernetes_apiserver [�[32mOK�[0m]
      Total Runs: 19
      Metric Samples: Last Run: 0, Total: 0
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    load
    ----
      Instance ID: load [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 6, Total: 120
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    memory
    ------
      Instance ID: memory [�[32mOK�[0m]
      Total Runs: 19
      Metric Samples: Last Run: 17, Total: 323
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
    
    network (1.9.0)
    ---------------
      Instance ID: network:2a218184ebe03606 [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 33, Total: 660
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 2ms
      
    
    ntp
    ---
      Instance ID: ntp:b4579e02d1981c12 [�[32mOK�[0m]
      Total Runs: 19
      Metric Samples: Last Run: 1, Total: 19
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 1, Total: 19
      Average Execution Time : 3ms
      
    
    uptime
    ------
      Instance ID: uptime [�[32mOK�[0m]
      Total Runs: 20
      Metric Samples: Last Run: 1, Total: 20
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      
========
JMXFetch
========

  Initialized checks
  ==================
    no checks
    
  Failed checks
  =============
    no checks
    
=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 19
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 2
    Metadata: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 40
    TimeseriesV1: 19

  API Keys status
  ===============
    API key ending with d4cbe: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.com - API Key ending with:
      - d4cbe

==========
Logs Agent
==========

  container_collect_all
  ---------------------
    Type: docker
    Status: Pending

=========
Aggregator
=========
  Checks Metric Sample: 7,215
  Dogstatsd Metric Sample: 48
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 19
  Series Flushed: 3,531
  Service Check: 255
  Service Checks Flushed: 262

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 48
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Packet Reading Errors: 0
  Udp Packets: 49
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://10.100.238.126:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: {Major:1 Minor:2 Patch:0 Pre: Meta: Commit:52053af}

Additional environment details (Operating System, Cloud provider, etc):
Kubernetes 1.12 cluster on EKS.

Steps to reproduce the issue:

1. Deploy the Datadog agent using the provider Kubernetes resources.
2. View logs

Describe the results you received:
https://gist.github.com/hden/9a947fd838d3e72c21f842ee8c4082bd
Many dashboard entries remain empty.

Describe the results you expected:
No errors, access to kubelet, functional Kubernetes dashboard.

Additional information you deem important (e.g. issue happens only occasionally):
Seems to be the same problem as #2582. Tried DD_KUBELET_TLS_VERIFY=false and DD_KUBERNETES_KUBELET_HOST -> spec.nodeName.