x509: cannot validate certificate for <WORKER_NODE_IP> because it doesn't contain any IP SANs

[MugenTwo]

My setup is:

1. Ubuntu server inside HyperV running MicroK8s (the host of the hyperv is the windows server). I followed the standard installation: https://microk8s.io/docs/getting-started
2. Windows server 2022 (which is the host of the hyperv that has the ubuntu server that has microk8s) running worker node installed following: https://microk8s.io/docs/add-a-windows-worker-node-to-microk8s
3. The cluster seems to be connected and I can run windows containers inside the windows worker node and linux containers in the ubuntu server microk8s (control plane + worker node).
4. This is what I see when I do a kubectl get nodes
   
5. However, there are cert issues whenever I do certain operations on pods running in the Windows worker node like kubectl log, kubectl port-forward:
   

Expected Behavior

There should NOT be cert errors.

Current Behavior

5. However, there are cert issues whenever I do certain operations on pods running in the Windows worker node like kubectl log, kubectl port-forward:
   

Possible Solution

I've tried to:

1. Initially I install Kubernetes services version 1.27.1, and I got the cert error.
2. So, I decided to uninstall this version 1.27.1 and then I installed 1.32.0 because my control plane (microk8s) had this version.
3. I tried deleting the cert: C:\var\lib\kubelet\pki and then restarting the kubelet.
4. I also checked how the kubelet was being run in the kubelet,out.log that is under "C:\k" and I printed the kubelet.exe parameters:
   
5. From what I read online the IP SAN might be dependant on the --node-ip, but the --node-ip in kubelet.exe args seems correct in my case.

None of which has fixed the issue

Steps to Reproduce (for bugs)

1. Ubuntu server inside HyperV running MicroK8s (the host of the hyperv is the windows server). I followed the standard installation: https://microk8s.io/docs/getting-started
2. Windows server 2022 (which is the host of the hyperv that has the ubuntu server that has microk8s) running worker node installed following: https://microk8s.io/docs/add-a-windows-worker-node-to-microk8s
3. Run any windows container in a pod.
4. Do a "kubectl log" on the pod.

Context

1. Ubuntu server inside HyperV running MicroK8s (the host of the hyperv is the windows server). I followed the standard installation: https://microk8s.io/docs/getting-started
2. Windows server 2022 (which is the host of the hyperv that has the ubuntu server that has microk8s) running worker node installed following: https://microk8s.io/docs/add-a-windows-worker-node-to-microk8s

Your Environment

• Calico version
  CALICO_VERSION="3.25.0"
• Calico dataplane (iptables, windows etc.)
• Orchestrator version (e.g. kubernetes, mesos, rkt):
• Operating System and version:
  Microk8s on ubuntu server running in HyperV (105 IP)
  Windows worker node (106 IP)
• Link to your project (optional):

[MugenTwo]

I'm linking this issue to the issue I created in MicroK8s: canonical/microk8s#4814
As I think it might have something to do with the integration with MicroK8s

[caseydavenport]

@MugenTwo what makes you believe this has something to do with Calico?

AFAIK, Calico isn't involved in the management of the certificate(s) in question. But my knowledge of the Windows side of things is a bit fuzzy. Perhaps @coutinhop can chime in here as well. But IMO this sounds like a problem with the way Kubernetes is installed on the cluster, and nothing to do with Calico.

[MugenTwo]

Hi @caseydavenport ! The reason is because the script to install and run kubelet and kube-proxy that is recommended in the Microk8s guide is done via calico scripts: https://github.com/projectcalico/calico/tree/master/node/windows-packaging/CalicoWindows/kubernetes

Here is the command from the Microk8s guide (https://microk8s.io/docs/add-a-windows-worker-node-to-microk8s):
c:\k\install-calico-windows.ps1 -ReleaseBaseURL "https://github.com/projectcalico/calico/releases/download/v3.25.0" -ReleaseFile "calico-windows-v3.25.0.zip" -KubeVersion "1.27.1" -DownloadOnly "yes" -ServiceCidr "10.152.183.0/24" -DNSServerIPs "10.152.183.10"

[coutinhop]

From finding #6754 and kubernetes/kubernetes#62939 (similar, but related to kubeadm configuration) it seems like it's a microk8s configuration issue... Perhaps best to follow up on canonical/microk8s#4814

[MugenTwo]

Hi @coutinhop ! I think I found a workaround to this. But I need to do another test, just to confirm (because I tried so many things and am unsure how I actually got it to work). But basically, I created the .crt file with the IP SANs and signed it with the CA.cert file from MicroK8s and replaced the certs in the kubelet of the windows worker node (the one installed by calico scripts). I will do another test sometime end of this month (or early next month) and comeback here with more details. Thanks! Appreciate your time!