kubeadm configmap not parsed properly when "certSANS" field contains IPv6 addresses in flow style

[cbf123]

What keywords did you search in kubeadm issues before filing this one?

IPv6, flow, certSANs

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version): Bug is present in at least 1.21 to 1.24

Environment:

• Kubernetes version (use kubectl version):
  sysadmin@controller-1:~$ kubectl version
  WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
  Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"archive", BuildDate:"2022-12-19T07:54:12Z", GoVersion:"go1.18.5", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.4", GitCommit:"95ee5ab382d64cfe6c28967f36b53970b8374491", GitTreeState:"clean", BuildDate:"2022-08-17T18:47:37Z", GoVersion:"go1.18.5", Compiler:"gc", P

• Cloud provider or hardware configuration:
  Intel server hardware

• OS (e.g. from /etc/os-release):
  sysadmin@controller-1:~$ cat /etc/os-release
  PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
  NAME="Debian GNU/Linux"
  VERSION_ID="11"
  VERSION="11 (bullseye)"
  VERSION_CODENAME=bullseye
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"

• Kernel (e.g. uname -a):
  sysadmin@controller-1:~$ uname -a
  Linux controller-1 5.10.0-6-amd64 kubeadm join on slave node fails preflight checks #1 SMP PREEMPT StarlingX Debian 5.10.162-1.stx.63 (2023-04-11) x86_64 GNU/Linux

• Container runtime (CRI) (e.g. containerd, cri-o):
  containerd

• Container networking plugin (CNI) (e.g. Calico, Cilium):
  Calico

What happened?

I ran the following commands and got errors as shown:

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get cm -n kube-system kubeadm-config -o=jsonpath={.data.ClusterConfiguration} > /tmp/asdf

sysadmin@controller-0:~$ kubeadm config migrate --old-config /tmp/asdf --new-config /tmp/asdf_new --v=6
could not interpret GroupVersionKind; unmarshal error: error converting YAML to JSON: yaml: line 1: did not find expected node content

After doing some experimenting, it seems like the problem only shows up if the "certSANS" field under "apiServer" contains unquoted IPv6 addresses, and is in "flow" style YAML.

The problematic problematic formatting looks like this:

        apiServer:
            certSANs: [::1, 192.168.206.1, 127.0.0.1, 10.20.7.3]

While this is fine:

        apiServer:
            certSANs:
            - ::1
            - 192.168.206.1
            - 127.0.0.1
            - 10.20.7.3

What you expected to happen?

I expected kubeadm to handle both IPv4 and IPv6 addresses the same, whether in flow style YAML or block style YAML

How to reproduce it (as minimally and precisely as possible)?

Dump the kubeadm configmap to a file as per above, edit the certSANs field to be in flow syle, if it contains IPv4 addresses then add "::1" as the first address. Then run "kubeadm config migrate" as per above.

Anything else we need to know?

After talking with the YAML developers in IRC chat, it seems that it is legal to have colons in flow mode as long as they don't have a space after the colon...but that not all of the YAML parsers handle this properly including Go yaml.

[neolit123]

After talking with the YAML developers in IRC chat, it seems that it is legal to have colons in flow mode as long as they don't have a space after the colon...but that not all of the YAML parsers handle this properly including Go yaml.

this seems like a problem with the yaml/json library kubeadm uses. it is also used by kubectl and core k8s!

there isn't much we can do here on the kubeadm side, but you could log an issue here:
https://github.com/kubernetes-sigs/yaml

and also provide a reproducible go playground example.

if for some reason this is only a kubeadm problem (unlikely), please comment again here.

[cbf123]

After doing some more digging, it looks like a problem with the go-yaml parser, as well as the libyaml parser that it was ported from. And it seems that the problem is strictly limited to the "::1" localhost address. Other IPv6 addresses like "feed:beef::1" work fine.

[cbf123]

I've opened kubernetes-sigs/yaml#92 and go-yaml/yaml#956

In the meantime, I guess the only workaround is to ensure that we always use block style when editing the kubeadm configmap for anything that might contain IPv6 addresses?

[cbf123]

Actually, after talking further about this specific use-case with the YAML devs, strictly speaking IPv6 addresses are not safe to include as unquoted strings in YAML because IPv6 addresses can validly end in a colon.

It looks like this is special-cased when running "kubeadm config migrate". If I start with

apiServer:
  certSANs: ['::1', 'dead:beef::', '1221:10a:a001:a103::1085']

and run "kubeadm config migrate" on it, then I end up with

apiServer:
  certSANs:
  - ::1
  - 'dead:beef::'
  - 1221:10a:a001:a103::1085

This will be safe to parse, but it's going to be really confusing for anyone looking at it since it's not clear at first glance why only one of the addresses is quoted. Maybe it would make sense to preserve all the quotes?

[neolit123]

Maybe it would make sense to preserve all the quotes?

this seems like a parser artifact too. i don't recall kubeadm code quoting ipv6 addresses during migrate.

certSANs: ['::1', 'dead:beef::', ....

note, while this is valid yaml, this is the first time i've seen a kubeadm user prepare the certsans like that. perhaps we should just note in kubeadm docs that they should not do that.

[cbf123]

Sorry, what do you mean by "prepare the certsans like that"? How should they be handled?

[neolit123]

Sorry, what do you mean by "prepare the certsans like that"? How should they be handled?

this is the first time i see certSANs in flow style yaml.