diff --git a/content/en/docs/setup/production-environment/container-runtimes.md b/content/en/docs/setup/production-environment/container-runtimes.md index 5aa4ac894e2e6..3f022d191f62c 100644 --- a/content/en/docs/setup/production-environment/container-runtimes.md +++ b/content/en/docs/setup/production-environment/container-runtimes.md @@ -48,7 +48,7 @@ Changing the settings such that your container runtime and kubelet use `systemd` stabilized the system. To configure this for Docker, set `native.cgroupdriver=systemd`. {{< caution >}} -Changing the cgroup driver of a Node that has joined a cluster is strongly *not* recommended. +Changing the cgroup driver of a Node that has joined a cluster is a sensitive operation. If the kubelet has created Pods using the semantics of one cgroup driver, changing the container runtime to another cgroup driver can cause errors when trying to re-create the Pod sandbox for such existing Pods. Restarting the kubelet may not solve such errors. @@ -57,6 +57,11 @@ If you have automation that makes it feasible, replace the node with another usi configuration, or reinstall it using automation. {{< /caution >}} +### Migrating to the `systemd` driver in kubeadm managed clusters + +Follow this [Migration guide](/tasks/administer-cluster/kubeadm/configure-cgroup-driver) +if you wish to migrate to the `systemd` cgroup driver in existing kubeadm managed clusters. + ## Container runtimes {{% thirdparty-content %}} diff --git a/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md b/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md index 394820324d7a7..74d6a79dca5c6 100644 --- a/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md +++ b/content/en/docs/setup/production-environment/tools/kubeadm/install-kubeadm.md @@ -233,7 +233,7 @@ sudo mkdir -p /opt/cni/bin curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" | sudo tar -C /opt/cni/bin -xz ``` -Define the directory to download command files +Define the directory to download command files {{< note >}} The `DOWNLOAD_DIR` variable must be set to a writable directory. @@ -284,33 +284,17 @@ See the [Kubeadm Troubleshooting guide](/docs/setup/production-environment/tools The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do. -## Configure cgroup driver used by kubelet on control-plane node +## Configuring a cgroup driver -When using Docker, kubeadm will automatically detect the cgroup driver for the kubelet -and set it in the `/var/lib/kubelet/config.yaml` file during runtime. +Both the container runtime and the kubelet have a property called +["cgroup driver"](/docs/setup/production-environment/container-runtimes/), which is important +for the management of cgroups on Linux machines. -If you are using a different CRI, you must pass your `cgroupDriver` value to `kubeadm init`, like so: - -```yaml -apiVersion: kubelet.config.k8s.io/v1beta1 -kind: KubeletConfiguration -cgroupDriver: -``` - -For further details, please read [Using kubeadm init with a configuration file](/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file). - -Please mind, that you **only** have to do that if the cgroup driver of your CRI -is not `cgroupfs`, because that is the default value in the kubelet already. - -{{< note >}} -Since `--cgroup-driver` flag has been deprecated by the kubelet, if you have that in `/var/lib/kubelet/kubeadm-flags.env` -or `/etc/default/kubelet`(`/etc/sysconfig/kubelet` for RPMs), please remove it and use the KubeletConfiguration instead -(stored in `/var/lib/kubelet/config.yaml` by default). -{{< /note >}} - -The automatic detection of cgroup driver for other container runtimes -like CRI-O and containerd is work in progress. +{{< warning >}} +Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail. +{{< warning >}} +See [Configuring a cgroup driver](/tasks/administer-cluster/kubeadm/configure-cgroup-driver) for more details. ## Troubleshooting diff --git a/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md new file mode 100644 index 0000000000000..02fdc958fa2b0 --- /dev/null +++ b/content/en/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver.md @@ -0,0 +1,127 @@ +--- +title: Configuring a cgroup driver +content_type: task +weight: 10 +--- + + + +This page explains how to configure the kubelet cgroup driver to match the container +runtime cgroup driver for kubeadm clusters. + +## {{% heading "prerequisites" %}} + +You should be familiar with the Kubernetes +[container runtime requirements](/docs/setup/production-environment/container-runtimes). + + + +## Configuring the container runtime cgroup driver + +The [Container runtimes](/docs/setup/production-environment/container-runtimes) page +explains that the `systemd` driver is recommended for kubeadm based setups instead +of the `cgroupfs` driver, because kubeadm manages the kubelet as a systemd service. + +The page also provides details on how to setup a number of different container runtimes with the +`systemd` driver by default. + +## Configuring the kubelet cgroup driver + +kubeadm allows you to pass a `KubeletConfiguration` structure during `kubeadm init`. +This `KubeletConfiguration` can include the `cgroupDriver` field which controls the cgroup +driver of the kubelet. + +{{< note >}} + +{{< feature-state for_k8s_version="v1.21" state="stable" >}} + +If the user is not setting the `cgroupDriver` field under `KubeletConfiguration`, +`kubeadm init` will default it to `systemd`. +{{< /note >}} + +A minimal example of configuring the field explicitly: + +```yaml +# kubeadm-config.yaml +kind: ClusterConfiguration +apiVersion: kubeadm.k8s.io/v1beta2 +kubernetesVersion: v1.21 +--- +kind: KubeletConfiguration +apiVersion: kubelet.config.k8s.io/v1beta1 +cgroupDriver: systemd +``` + +Such a configuration file can then be passed to the kubeadm command: + +```shell +kubeadm init --config kubeadm-config.yaml +``` + +{{< note >}} +Kubeadm uses the same `KubeletConfiguration` for all nodes in the cluster. +The `KubeletConfiguration` is stored in a [ConfigMap](docs/concepts/configuration/configmap) +object under the `kube-system` namespace. + +Executing the sub commands `init`, `join` and `upgrade` would result in kubeadm +writing the `KubeletConfiguration` as a file under `/var/lib/kubelet/config.yaml` +and passing it to the local node kubelet. +{{< /note >}} + +## Using the `cgroupfs` driver + +As this guide explains using the `cgroupfs` driver with kubeadm is not recommended. + +To continue using `cgroupfs` and to prevent `kubeadm upgrade` from modifying the +`KubeletConfiguration` cgroup driver on existing setups, you must be explicit +about its value. This applies to a case where you do not wish future versions +of kubeadm to apply the `systemd` driver by default. + +See the below section on "Modify the kubelet ConfigMap" for details on +how to be explicit about the value. + +If you wish to configure a container runtime to use the `cgroupfs` driver, +you must refer to the documentation of the container runtime of your choice. + +## Migrating to the `systemd` driver + +To change the cgroup driver of an existing kubeadm cluster to `systemd` in-place, +a similar procedure to a kubelet upgrade is required. This must include both +steps outlined below. + +{{< note >}} +Alternatively, it is possible to replace the old nodes in the cluster with new ones +that use the `systemd` driver. This requires executing only the first step below +before joining the new nodes and ensuring the workloads can safely move to the new +nodes before deleting the old nodes. +{{< /note >}} + +### Modify the kubelet ConfigMap + +- Find the kubelet ConfigMap name using `kubectl get cm -n kube-system | grep kubelet-config`. +- Call `kubectl edit cm kubelet-config-x.yy -n kube-system` (replace `x.yy` with +the Kubernetes version). +- Either modify the existing `cgroupDriver` value or add a new field that looks like this: + + ```yaml + cgroupDriver: systemd + ``` + This field must be present under the `kubelet:` section of the ConfigMap. + +### Update the cgroup driver on all nodes + +For each node in the cluster: + +- [Drain the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl drain --ignore-daemonsets` +- Stop the kubelet using `systemctl stop kubelet` +- Stop the container runtime +- Modify the container runtime cgroup driver to `systemd` +- Set `cgroupDriver: systemd` in `/var/lib/kubelet/config.yaml` +- Start the container runtime +- Start the kubelet using `systemctl start kubelet` +- [Uncordon the node](/docs/tasks/administer-cluster/safely-drain-node) using `kubectl uncordon ` + +Execute these steps on nodes one at a time to ensure workloads +have sufficient time to schedule on different nodes. + +Once the process is complete ensure that all nodes and workloads are healthy.