docker is required for container runtime even though I am using containerd #2364

brianmay · 2020-12-15T01:05:39Z

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT

Versions

kubeadm version (use kubeadm version):

root@kube-master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:57:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version): 0.20.0
OS (e.g. from /etc/os-release): Debian/buster
Kernel (e.g. uname -a): 4.19.0-13-amd64

What happened?

kubadm upgrade node tries to run docker, but I have switched to containerd:

root@kube-master:~# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.0", GitCommit:"af46c47ce925f4c4ad5cc8d1fca46c7b77d13b38", GitTreeState:"clean", BuildDate:"2020-12-08T17:57:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
root@kube-master:~# kubeadm config images pull 
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.20.0
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.20.0
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.20.0
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.20.0
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0
[config/images] Pulled k8s.gcr.io/coredns:1.7.0
root@kube-master:~# kubeadm upgrade node 
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W1215 11:40:34.109125   26843 kubelet.go:200] cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: docker is required for container runtime: exec: "docker": executable file not found in $PATH
To see the stack trace of this error execute with --v=5 or higher

What you expected to happen?

"kube upgrade node" like "kubeadm config images pull" should run cri commands, not docker commands.

I think the "docker info" part is related to #2270 - but in that case it is a warning only.

But It looks like the last message is a hard error.

The text was updated successfully, but these errors were encountered:

neolit123 · 2020-12-15T01:50:37Z

your workaround is to skip the phase for now --skip-phases=preflight

kubeadm config images pull

i just tried removing the docker binary and this command fails for me too.
kubeadm config images pull has no logic to detect if you wish to use crictl (containerd) unless you pass --config (with the socket value) or --cri-socket. i don't see how this command is passing in your case and you should be getting the same error

"docker": executable file not found in $PATH

kubeadm upgrade node

the kubeadm upgrade node currently has no way to accept a CRI socket (e.g. via --cri-socket), so it currently just defaults to the docker socket and therefore defaults to using the docker CLI.

one solution here is to fetch what CRI socket is on this Node object, but this means we need to know the node name.

kubectl get no controlplane -o yaml | grep cri
    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock

the alternative is to require the user to pass --cri-socket if they want to use a container runtime != docker.

neolit123 · 2020-12-15T01:59:08Z

we did remove the --cri-socket flag for upgrade apply with the argument it should be fetched from the cluster:
kubernetes/kubernetes#85044
#1356

so it seems appropriate to fetch it from the Node object.

cc @fabriziopandini @SataQiu WDYT?

BTW @SataQiu looks like this wasn't a sufficient fix:
kubernetes/kubernetes#94555

... or instead of fetching the Node cri-socket, we may have to apply CRI socket detection here:
https://github.com/kubernetes/kubernetes/blob/89ba90573f163ee3452b526f30348a035d54e870/cmd/kubeadm/app/cmd/upgrade/node.go#L148
and here:
https://github.com/kubernetes/kubernetes/blob/0b92e8b16d00712594493072710f81b8d37ce623/cmd/kubeadm/app/cmd/upgrade/common.go#L76

/kind feature
/area ux

neolit123 · 2020-12-15T02:02:30Z

@brianmay i'd assume this is a problem for upgrade apply too?
(except that upgrade apply allows passing an InitConfiguration.NodeRegistrationOptions.CRISocket via --config)

brianmay · 2020-12-15T03:30:24Z

Yes, this is on upgrades. As above, looks like there might be a work around via the --config parameter. Will try asap.

brianmay · 2020-12-15T03:44:30Z

Sorry, ignore my previous response. I was getting confused.

What is the difference between "upgrade apply" and "upgrade node"? Can I pass a config file that only sets CRISocket or do I need to set all the other values too?

rsoika · 2020-12-29T21:16:24Z

I have the same issue when calling the print-join-command.
I installed kubernetes v1.20.1 on Debian with containerd. THe master and worker nodes work.
But if i call on the master node:

kubeadm token create --print-join-command

I got:

W1229 21:59:35.779624   18425 kubelet.go:200] cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH
kubeadm join 10.0.0.2:6443 --token 4fd7i0.v8kddn2yfrgtp544     --discovery-token-ca-cert-hash sha256:.........

I did not really understand all the discussion about this warning. Should we ignore this? Joining a worker node to the master is working fine - even without docker daemon installed. And the cluster seems to work.

brianmay · 2020-12-30T01:41:50Z

@rsoika I believe that warning can be ignored. It is the hard error I was getting that cannot be ignored.

I am hoping I might be able to resolve this without converting my cluster back to docker... But so far everyone seems to be rather quiet on the subject of a solution or even a workaround.

Unless of course kubeadm 1.20.1 has made any changes to fix this?

neolit123 · 2020-12-30T18:20:30Z

we should fix this after the holidays.

brianmay · 2020-12-30T23:13:06Z

@neolit123 Great news, thanks.

pacoxu · 2020-12-31T03:06:05Z

after I remove /var/run/docker-shim.sock and /var/run/docker.sock, the command works.

neolit123 · 2020-12-31T03:09:07Z

removing /var/run/docker*.sock is actually a good solution. when no config file is passed to a command kubeadm (with an explicit socket) and if the docker socket is present on the host it will take priority.

brianmay · 2020-12-31T10:15:57Z

In my case I don't have anything that matches /var/run/docker*, or anything that looks like these sock files. I think I might have deleted them already.

I do have a /var/lib/docker and a /var/lib/dockershim/ (it is safe to delete these??) but I am a little bit skeptical these directories are confusing kubeadm.

AleksandrNull · 2021-01-04T00:56:26Z

Here is a KISS workaround:
echo '#!/bin/sh' > /sbin/docker && chmod 0100 /sbin/docker
and run kubeadm upgrade apply or kubeadm upgrade node command. Don't forget to delete /sbin/docker after upgrade was completed :)

brianmay · 2021-01-04T21:48:09Z

@AleksandrNull So I guess this means that the docker call's aren't actually required for the upgrade to work? If so, good to know.

jeanluclariviere · 2021-01-04T22:11:10Z

@AleksandrNull So I guess this means that the docker call's aren't actually required for the upgrade to work? If so, good to know.

I was hitting this exact error when trying to use kubeadm upgrade apply. @AleksandrNull solution worked perfectly and I was able to upgrade my dev cluster to 1.19.5 this morning.

AleksandrNull · 2021-01-05T15:26:45Z

@brianmay That's correct. It basically checks docker binary and trying to pre-pull images. Pulling images (using docker) is absolutely useless with containerd runtime as default so this "mock" does no harm.

neolit123 · 2021-01-08T12:17:28Z

@brianmay i tested and looked at the code today, it looks fine.

my guess is that you switched to containerd but the CRI socket on that Node object remains for docker.
what is the output of this command on that particular Node?

kubectl get no controlplane -o yaml | grep kubeadm.alpha.kubernetes.io/cri-socket

if you patch/edit the kubeadm.alpha.kubernetes.io/cri-socket value the kubeadm command should work.

kubeadm does not really support switching container runtimes on the fly or similar reconfiguration during upgrade...
we are in the process of writing guides around in-place container runtime replacement.

please check this discussion:
kubernetes/website#25787 (review)

and watch this ticket:
kubernetes/website#25879

neolit123 · 2021-01-08T12:25:14Z

this PR should make all commands that don't need the container runtime to not check for running docker or crtctl:
kubernetes/kubernetes#97625

neolit123 · 2021-01-08T12:32:18Z

@pacoxu would you have time to backport your PR to 1.18, 1.19, 1.20?

pacoxu · 2021-01-08T12:38:18Z

@neolit123 ok let me do it

brianmay · 2021-01-08T22:56:18Z

So for every control plane node I get:

$ kubectl get no kube-master -o yaml | grep kubeadm.alpha.kubernetes.io/cri-socket
kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock

Can I confirm that this - or similar- is the correct command to fix (for every control plane node):

kubectl annotate node master kubeadm.alpha.kubernetes.io/cri-socket unix:///run/containerd/containerd.sock

If I had known that you were still writing migration documentation, I might have waited. It is perhaps unfortunate that docker-shim was announced as deprecated, you should migrate over, etc, before the documentation was complete. And often projects don't bother with upgrade instructions :-(.

But regardless, thanks for the references supplied above to the PR and issue.

For the record the migration was relatively straight forward. Nothing on my system depends on Docker. Except the CNI file, which has somewhat painful to work out, particularly as I am using dual IPv4 and IPv6, and need to supply multiple subnets. Supposedly the auto-generated file was suppose to appear in my logs from before the migration, but I looked and looked and couldn't find it. I think I worked it out, but my solution does involve hard coding the nodes' subnet ranges. IIRC I tried "usePodCidr" for the IPv4 subnet and got loud objections. Would be nice if I didn't have to do this. But is acceptable for this cluster.

 {
	"cniVersion": "0.3.1",
	"name": "kubenet",
  	"type": "bridge",
    	"bridge": "cbr0",
    	"mtu": 1500,
    	"isGateway": true,
    	"ipMasq": true,
    	"hairpinMode": false,
    	"ipam": {
  		"type": "host-local",
       		"ranges": [
       		   [
       		       { "subnet": "10.1.0.0/16" }
       		   ],
       		   [
       		       { "subnet": "fc00:1::/32" }
       		   ]
       		],
        	"routes": [
			{ "dst": "0.0.0.0/0" },
			{ "dst": "::/0" }
		]
    	}
}

neolit123 · 2021-01-09T00:15:59Z

If I had known that you were still writing migration documentation, I might have waited. It is perhaps unfortunate that docker-shim was announced as deprecated, you should migrate over, etc, before the documentation was complete.

i warned about this on the deprecation PR:
kubernetes/kubernetes#94624 (comment)

neolit123 · 2021-01-09T00:22:07Z

kubectl annotate node master kubeadm.alpha.kubernetes.io/cri-socket unix:///run/containerd/containerd.sock

we should include this in the migration guide (TBD).

For the record the migration was relatively straight forward. Nothing on my system depends on Docker. Except the CNI file, which has somewhat painful to work out, particularly as I am using dual IPv4 and IPv6, and need to supply multiple subnets. Supposedly the auto-generated file was suppose to appear in my logs from before the migration, but I looked and looked and couldn't find it. I think I worked it out, but my solution does involve hard coding the nodes' subnet ranges. IIRC I tried "usePodCidr" for the IPv4 subnet and got loud objections. Would be nice if I didn't have to do this. But is acceptable for this cluster.

i didn't have to do this when i tried migrating docker -> containerd but it was a single stack v4.
we can consult with SIG Network if this becomes a common issue.

closing as the main issue is explained and the side issues were addressed by PR.
/close

k8s-ci-robot · 2021-01-09T00:22:20Z

@neolit123: Closing this issue.

In response to this:

kubectl annotate node master kubeadm.alpha.kubernetes.io/cri-socket unix:///run/containerd/containerd.sock

we should include this in the migration guide (TBD).

For the record the migration was relatively straight forward. Nothing on my system depends on Docker. Except the CNI file, which has somewhat painful to work out, particularly as I am using dual IPv4 and IPv6, and need to supply multiple subnets. Supposedly the auto-generated file was suppose to appear in my logs from before the migration, but I looked and looked and couldn't find it. I think I worked it out, but my solution does involve hard coding the nodes' subnet ranges. IIRC I tried "usePodCidr" for the IPv4 subnet and got loud objections. Would be nice if I didn't have to do this. But is acceptable for this cluster.

i didn't have to do this when i tried migrating docker -> containerd but it was a single stack v4.
we can consult with SIG Network if this becomes a common issue.

closing as the main issue is explained and the side issues were addressed by PR.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

brianmay · 2021-01-09T00:51:45Z

Revised the above command:

kubectl annotate node kube-master --overwrite kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock

It looks good now.

KeithTt · 2022-08-10T06:25:00Z

Revised the above command:

kubectl annotate node kube-master --overwrite kubeadm.alpha.kubernetes.io/cri-socket=unix:///run/containerd/containerd.sock

It looks good now.

I think this command need to communicate with apiserver, if apiserver is stopped how to update the value of kubeadm.alpha.kubernetes.io/cri-socket.

I refer to this guide: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/

stop kubelet
configure and start containerd
configure kubelet to use containerd
- update the file /var/lib/kubelet/kubeadm-flags.env
- kubectl edit no <node-name> to change the value of kubeadm.alpha.kubernetes.io/cri-socket from /var/run/dockershim.sock to /var/run/containerd/containerd.sock

The problem is that the kubelet is already stopped, and the apiserver pod is also stopped, I can't run kubectl edit no <node-name> to update the node infomation.

neolit123 · 2022-08-10T09:16:51Z

The problem is that the kubelet is already stopped, and the apiserver pod is also stopped, I can't run kubectl edit no to update the node infomation.

for single control plane clusters this can be a problem, yes. you can log an issue in kubernetes/website about it. the annotation can be safely edited before the kubelet is stopped.

KeithTt · 2022-08-10T10:13:08Z

the annotation can be safely edited before the kubelet is stopped.

Thanks a loooooot. Confused for a few days.

aimcod · 2023-04-11T10:36:20Z

I created this script to migrate from docker to containerd. This works on oracle linux and rocky linux.
Note the upper to lower converstion of the hostname, as kubernetes does that.

kubectl drain $(hostname | tr '[:upper:]' '[:lower:]') --ignore-daemonsets &
echo waiting 30 seconds for node $(hostname) to be drained...
sleep 30
sudo systemctl stop kubelet
sudo systemctl disable docker --now
sudo yum remove docker-ce docker-ce-cli -y
sudo modprobe overlay
sudo modprobe br_netfilter
sudo cat << EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
sudo cat << EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF
sudo sysctl --system
sudo yum install containerd -y
sudo mkdir  -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo sed -i 's/disabled_plugins.*/#disabled_plugins/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo sed -i.bak 's/KUBELET_KUBEADM_ARGS=\".*/KUBELET_KUBEADM_ARGS=\"--container-runtime=remote  --container-runtime-endpoint=unix:\/\/\/run\/containerd\/containerd.sock\"/' /var/lib/kubelet/kubeadm-flags.env

kubectl patch no  $(hostname | tr '[:upper:]' '[:lower:]') --patch '{"metadata": {"annotations": {"kubeadm.alpha.kubernetes.io/cri-socket": "unix:///run/containerd/containerd.sock"}}}'

sudo systemctl start kubelet

kubectl uncordon  $(hostname | tr '[:upper:]' '[:lower:]')

kubectl get nodes -o wide

This has worked for me, these past few days, as I am working on automating this task, as well as upgrading the entire system to rocky linux 9 along with kubernetes, to the latest version.

I think that due to this post, I finally know why my kubernetes upgrades were failing.
Big thanks to @AleksandrNull

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. area/UX labels Dec 15, 2020

neolit123 added this to the v1.21 milestone Dec 15, 2020

pacoxu mentioned this issue Dec 31, 2020

kubeadm: avoid detection of the container runtime for commands that do not need it kubernetes/kubernetes#97625

Merged

rsoika mentioned this issue Jan 4, 2021

Change from docker to containerd imixs/imixs-cloud#57

Closed

2 tasks

pacoxu mentioned this issue Jan 4, 2021

kubeadm error log kubernetes/kubernetes#97679

Closed

neolit123 mentioned this issue Jan 6, 2021

kubeadm join/init [WARNING IsDockerSystemdCheck] , when Use containerd. #2374

Closed

k8s-ci-robot closed this as completed Jan 9, 2021

pacoxu mentioned this issue Jan 24, 2021

"kubeadm init" does not auto-detect CRI-O as documentation states kubernetes/kubernetes#98335

Closed

neolit123 mentioned this issue Feb 13, 2021

Cannot automatically set CgroupDriver when starting the Kubelet: cannot execute 'docker info -f {{.CgroupDriver}}': executable file not found in $PATH kubernetes/kubernetes#99053

Closed

emosbaugh mentioned this issue Apr 11, 2023

fix(container): migration from docker on remove nodes fails with downgrade not supported message replicatedhq/kURL#4366

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker is required for container runtime even though I am using containerd #2364

docker is required for container runtime even though I am using containerd #2364

brianmay commented Dec 15, 2020

neolit123 commented Dec 15, 2020 •

edited

Loading

neolit123 commented Dec 15, 2020 •

edited

Loading

neolit123 commented Dec 15, 2020 •

edited

Loading

brianmay commented Dec 15, 2020

brianmay commented Dec 15, 2020

rsoika commented Dec 29, 2020

brianmay commented Dec 30, 2020 •

edited

Loading

neolit123 commented Dec 30, 2020

brianmay commented Dec 30, 2020

pacoxu commented Dec 31, 2020

neolit123 commented Dec 31, 2020 •

edited

Loading

brianmay commented Dec 31, 2020

AleksandrNull commented Jan 4, 2021 •

edited

Loading

brianmay commented Jan 4, 2021

jeanluclariviere commented Jan 4, 2021

AleksandrNull commented Jan 5, 2021

neolit123 commented Jan 8, 2021 •

edited

Loading

neolit123 commented Jan 8, 2021

neolit123 commented Jan 8, 2021 •

edited

Loading

pacoxu commented Jan 8, 2021

brianmay commented Jan 8, 2021

neolit123 commented Jan 9, 2021

neolit123 commented Jan 9, 2021

k8s-ci-robot commented Jan 9, 2021

brianmay commented Jan 9, 2021

KeithTt commented Aug 10, 2022

neolit123 commented Aug 10, 2022

KeithTt commented Aug 10, 2022

aimcod commented Apr 11, 2023

docker is required for container runtime even though I am using containerd #2364

docker is required for container runtime even though I am using containerd #2364

Comments

brianmay commented Dec 15, 2020

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

neolit123 commented Dec 15, 2020 • edited Loading

neolit123 commented Dec 15, 2020 • edited Loading

neolit123 commented Dec 15, 2020 • edited Loading

brianmay commented Dec 15, 2020

brianmay commented Dec 15, 2020

rsoika commented Dec 29, 2020

brianmay commented Dec 30, 2020 • edited Loading

neolit123 commented Dec 30, 2020

brianmay commented Dec 30, 2020

pacoxu commented Dec 31, 2020

neolit123 commented Dec 31, 2020 • edited Loading

brianmay commented Dec 31, 2020

AleksandrNull commented Jan 4, 2021 • edited Loading

brianmay commented Jan 4, 2021

jeanluclariviere commented Jan 4, 2021

AleksandrNull commented Jan 5, 2021

neolit123 commented Jan 8, 2021 • edited Loading

neolit123 commented Jan 8, 2021

neolit123 commented Jan 8, 2021 • edited Loading

pacoxu commented Jan 8, 2021

brianmay commented Jan 8, 2021

neolit123 commented Jan 9, 2021

neolit123 commented Jan 9, 2021

k8s-ci-robot commented Jan 9, 2021

brianmay commented Jan 9, 2021

KeithTt commented Aug 10, 2022

neolit123 commented Aug 10, 2022

KeithTt commented Aug 10, 2022

aimcod commented Apr 11, 2023

neolit123 commented Dec 15, 2020 •

edited

Loading

neolit123 commented Dec 15, 2020 •

edited

Loading

neolit123 commented Dec 15, 2020 •

edited

Loading

brianmay commented Dec 30, 2020 •

edited

Loading

neolit123 commented Dec 31, 2020 •

edited

Loading

AleksandrNull commented Jan 4, 2021 •

edited

Loading

neolit123 commented Jan 8, 2021 •

edited

Loading

neolit123 commented Jan 8, 2021 •

edited

Loading