Creating a v1.27.2 cluster on swap enabled host fails

[chrischdi]

What steps did you take and what happened?

• Run tilt-setup with CAPD
• Create cluster with Kubernetes v1.27.2
• See the kubelet not starting due to active swap

What did you expect to happen?

• kubelet runs without issues

Cluster API version

main

Kubernetes version

v1.27.2

Anything else you would like to add?

CI is good because there's no swap there.

For kind node images <= v1.27.2, the fail-swap-on flag for the kubelet was always said.
This flag got removed in https://github.com/kubernetes-sigs/kind/pull/3240/files in favor of setting it via the KubeletConfiguration, which kind was already doing.

There are multiple ways to fix this for CAPD:

Technically it comes down to either:

• Setting the --fail-swap-on=false flag on the kubelet (the flag itself is deprecated for usage, I don't know when it gets removed)
• Adding a KubeletConfiguration and configuring the failSwapOn option (like kind does here)

And there are multiple ways to do that for CAPD, high-level:

• Adjust the ClusterClass (templates or add a json patch)
• Inject the setting in some way in go code, but only for CAPD

Label(s) to be applied

/kind bug

[killianmuldoon]

/triage accepted

Thanks for filing this - I'm probably in favor of adding an inline patch to manage this for our CAPD templates as that's the way we tended to handle issues like this in the past - e.g. cgroup stuff.

I don't think we should worry too much about the arg being deprecated until we have some idea about when it will be removed.

[chrischdi]

I totally agree on this.

I can file that PR to update if we agree on doing it that way: https://github.com/kubernetes-sigs/cluster-api/blob/main/test/infrastructure/docker/templates/clusterclass-quick-start.yaml and test/e2e/data/infrastructure-docker/<version>/clusterclass-quick-start.yaml.

Two options there:

• ClusterClass JSON Patch (similar to cgroupDriver patches)
• Directly at the kubeletExtraArgs in KubeadmConfigTemplate and KubeadmControlPlaneTemplate (similar to eviction-hard argument)

[killianmuldoon]

Let's do the ClusterClass JSON Patch IMO

[killianmuldoon]

Should probably update our other ClusterClasses - those used for e2e tests etc. Even though we don't see this error in CI running those locally probably doesn't work right now.

[chrischdi]

Let's do the ClusterClass JSON Patch IMO

👍 on that.

Just wanted to add that we can make it version dependent on the first variant, so we don't break possible old versions which don't have the flag.

[chrischdi]

/assign

[chrischdi]

Note: feature got introduced as alpha in kubernetes v1.22 kubernetes/enhancements#2400

[sbueringer]

+1 to going with a ClusterClass JSON patch + keeping our ClusterClasses as similar as possible

Potentially we can replace this later if kubernetes/kubeadm#1682 is implemented before the command line flag is dropped.