Windows failed to create kubelet: failed to get docker version

[jsturtevant]

Is this a request for help?:
no

Is this an ISSUE or FEATURE REQUEST? (choose one):
issue

What version of acs-engine?:
any using windows server 1803 image

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)
Kubernetes

What happened:
deployed 1803 windows container images. Windows nodes never join cluster. get the error as described here: kubernetes/kubernetes#69996

server.go:262] failed to run Kubelet: failed to create kubelet: failed to get docker version: Error response from daemon: client version 1.38 is too new. Maximum supported API version is 1.37

What you expected to happen:
Kubelet to start and and connect nodes to the cluster

How to reproduce it (as minimally and precisely as possible):
deploy a k8s cluster with 1803 node as of today

Anything else we need to know:
Because the Docker version is installed by the image the version got auto upgraded. Should pin the Docker version to a known working value so that Windows Server moving to new Docker version upgrades in Azure don't cause deployment errors.

[zhiweiv]

Today I created a new k8s cluster due to the old one crashed #4046, but the Windows nodes didn't appear in "kubectl get nodes", although the Windows VMs were created in Azure.

Do we have workaround? It is a blocking issue, because we are unable to run any workloads in k8s now.

Thanks

[jsturtevant]

The PR linked above fixes it. Hoping to get that merged soon. As a stop gap, if you remote into each machine and set the docker version manually to 17.06.2-ee-16: #3852 (comment)

[zhiweiv]

@PatrickLang @jsturtevant
0.24.2 still does not work, could you please take a look again?

The latest kubelet.err.log:
I1026 10:55:57.715279 96948 server.go:408] Version: v1.12.1
E1026 10:55:58.464121 96948 server.go:731] Kubelet needs to run as uid 0. It is being run as -1
W1026 10:55:58.464121 96948 server.go:741] setting OOM scores is unsupported in this build
I1026 10:55:58.464121 96948 kubelet.go:304] Watching apiserver
W1026 10:55:58.469042 96948 iptables.go:155] Error checking iptables version, assuming version at least 1.4.11: executable file not found in %PATH%
I1026 10:55:58.472042 96948 client.go:104] Start docker client with request timeout=2m0s
E1026 10:55:58.479109 96948 kube_docker_client.go:91] failed to retrieve docker version: Error response from daemon: client version 1.38 is too new. Maximum supported API version is 1.37
W1026 10:55:58.479109 96948 kube_docker_client.go:92] Using empty version for docker client, this may sometimes cause compatibility issue.
F1026 10:55:58.479109 96948 server.go:262] failed to run Kubelet: failed to create kubelet: failed to get docker version: Error response from daemon: client version 1.38 is too new. Maximum supported API version is 1.37

Thanks

[zhiweiv]

Seems this PR is not included in 0.24.2?

[jsturtevant]

I don't beleive it is, as 0.24.2 was a patch release. The change is in master so next release should be avaliable or you can build from master.

[PatrickLang]

As a temporary workaround, you can also deploy the previous Windows image by setting imageVersion like this. It will work with existing acs-engine releases including 0.24.2.

    "windowsProfile": {
                                "adminUsername": "...",
                                "adminPassword": "...",
                                "windowsPublisher": "MicrosoftWindowsServer",
                                "windowsOffer": "WindowsServerSemiAnnual",
                                "windowsSku": "Datacenter-Core-1803-with-Containers-smalldisk",
                                "imageVersion": "1803.0.20180912"
                        },

[zhiweiv]

The workaround works,

Thanks

[GLink]

Should pinning the image be the norm, not just a workaround? This issue would also have happened on scale operations on working clusters, right?

[zhiweiv]

The scale does not work also. I met this issue first when scaling my cluster.

[PatrickLang]

Docker version is pinned now in 0.24.3, but you can set it it the API model if you want a different version.