The fully-private cluster created with eksctl 0.48.0 does not work, but everything works fine with eksctl 0.38.0

[yufan022]

What help do you need?

Create a fully-private cluster by eksctl 0.48.0, nodes can't join the cluster.
But I use the same configuration to create a cluster by eksctl 0.38.0, everything is ok.

What's the difference between the two versions?
Why can't the nodes work in the cluster created by ekscl 0.48.0?

# https://github.com/weaveworks/eksctl/tree/main/examples
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: eks-tidb3
  region: ap-northeast-1
  # version: "1.18"
  version: "1.19"

vpc:
  id: "vpc-xxx"
  securityGroup: "sg-xxx"
  sharedNodeSecurityGroup: "sg-xxx"
  subnets:
    private:
      ap-northeast-1c:
          id: "subnet-xxx"
      ap-northeast-1a:
          id: "subnet-xxx"
      ap-northeast-1d:
          id: "subnet-xxx"
  # clusterEndpoints:
  #   publicAccess: false
  #   privateAccess: true


iam:
  withOIDC: false

privateCluster:
  enabled: true
  additionalEndpointServices:
  - "autoscaling"
  - "logs"
  - "cloudformation"


nodeGroups:
  - name: tidb-1d
    # instanceType: m5.large
    desiredCapacity: 1
    instanceType: c5.xlarge
    privateNetworking: true
    # availabilityZones: ["ap-northeast-1d"]
    subnets:
      - ap-northeast-1c
    labels:
      dedicated: tidb
    ssh:
      allow: true
      publicKeyPath: "~/.ssh/id_rsa.pub"
      sourceSecurityGroupIds: ["sg-xxx"]
    securityGroups:
      attachIDs: ["sg-xxx"]

[root@ip-10-0-1-46 ~]# eksctl version
0.48.0
[root@ip-10-0-1-46 ~]# ./eksctl version
0.38.0

[root@ip-10-0-1-46 ~]# kubectl get nodes
No resources found in default namespace.

nodes

[root@ip-10-0-30-120 ec2-user]# ps -ef | grep eks
root      4369  4357  0 06:57 ?        00:00:00 /bin/bash /var/lib/cloud/scripts/eksctl/bootstrap.al2.sh
root      4376  4369  0 06:57 ?        00:00:00 bash /etc/eks/bootstrap.sh eks-tidb3 --dns-cluster-ip  --kubelet-extra-args --register-with-taints= --node-labels=dedicated=tidb,alpha.eksctl.io/cluster-name=eks-tidb3,alpha.eksctl.io/nodegroup-name=tidb-1d,node-lifecycle=on-demand,alpha.eksctl.io/instance-id=i-0fe9673f2de26f48a
root      4416  4376  0 06:57 ?        00:00:00 /usr/bin/python2 -s /usr/bin/aws eks wait cluster-active --region=ap-northeast-1 --name=eks-tidb3
root      4838  4779  0 07:03 pts/0    00:00:00 grep --color=auto eks

2021-05-20 06:33:25 [ℹ]  eksctl version 0.48.0
2021-05-20 06:33:25 [ℹ]  using region ap-northeast-1
2021-05-20 06:33:25 [!]  security group rules may be added by eksctl; see vpc.manageSharedNodeSecurityGroupRules to disable this behavior
2021-05-20 06:33:26 [✔]  using existing VPC (vpc-xx) and subnets (private:map[ap-northeast-1a:{subnet-xx ap-northeast-1a 10.0.2.0/24} ap-northeast-1c:{subnet-xx ap-northeast-1c 10.0.16.0/20} ap-northeast-1d:{subnet-xx ap-northeast-1d 10.0.3.0/24}] public:map[])
2021-05-20 06:33:26 [!]  custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets
2021-05-20 06:33:26 [ℹ]  nodegroup "tidb-1d" will use "ami-xx" [AmazonLinux2/1.19]
2021-05-20 06:33:26 [ℹ]  using SSH public key "/root/.ssh/id_rsa.pub" as "eksctl-eks-tidb3-nodegroup-tidb-1d-d4:87:f4:51:4f:ea:9f:2a:cc:04:21:f4:4e:xx:xx:xx"
2021-05-20 06:33:26 [ℹ]  using Kubernetes version 1.19
2021-05-20 06:33:26 [ℹ]  creating EKS cluster "eks-tidb3" in "ap-northeast-1" region with un-managed nodes
2021-05-20 06:33:26 [ℹ]  1 nodegroup (tidb-1d) was included (based on the include/exclude rules)
2021-05-20 06:33:26 [ℹ]  will create a CloudFormation stack for cluster itself and 1 nodegroup stack(s)
2021-05-20 06:33:26 [ℹ]  will create a CloudFormation stack for cluster itself and 0 managed nodegroup stack(s)
2021-05-20 06:33:26 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-1 --cluster=eks-tidb3'
2021-05-20 06:33:26 [ℹ]  CloudWatch logging will not be enabled for cluster "eks-tidb3" in "ap-northeast-1"
2021-05-20 06:33:26 [ℹ]  you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-1 --cluster=eks-tidb3'
2021-05-20 06:33:26 [ℹ]  Kubernetes API endpoint access will use provided values {publicAccess=true, privateAccess=true} for cluster "eks-tidb3" in "ap-northeast-1"
2021-05-20 06:33:26 [ℹ]  2 sequential tasks: { create cluster control plane "eks-tidb3", 3 sequential sub-tasks: { 2 sequential sub-tasks: { wait for control plane to become ready, update cluster VPC endpoint access configuration }, create addons, create nodegroup "tidb-1d" } }
2021-05-20 06:33:26 [ℹ]  building cluster stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:33:26 [ℹ]  deploying stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:33:56 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:34:27 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:35:27 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:36:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:37:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:38:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:39:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:40:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:41:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:42:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:43:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:44:37 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-cluster"
2021-05-20 06:44:40 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:44:56 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:45:13 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:45:33 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:45:50 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:46:10 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:46:29 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:46:48 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:47:05 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:47:23 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:47:39 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:47:56 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:48:14 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:48:30 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:48:46 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:49:03 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:49:19 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:49:37 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:49:54 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:50:13 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:50:30 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:50:49 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:51:06 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:51:26 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:51:42 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:51:58 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:52:15 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:52:31 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:52:48 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:53:08 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:53:24 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:53:43 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "eks-tidb3" to succeed
2021-05-20 06:53:44 [ℹ]  building nodegroup stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:53:44 [ℹ]  --nodes-min=1 was set automatically for nodegroup tidb-1d
2021-05-20 06:53:44 [ℹ]  --nodes-max=1 was set automatically for nodegroup tidb-1d
2021-05-20 06:53:44 [ℹ]  deploying stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:53:44 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:54:03 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:54:18 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:54:35 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:54:53 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:55:10 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:55:28 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:55:44 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:56:04 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:56:23 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:56:40 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:56:56 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:57:14 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:57:29 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:57:46 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:58:05 [ℹ]  waiting for CloudFormation stack "eksctl-eks-tidb3-nodegroup-tidb-1d"
2021-05-20 06:58:06 [ℹ]  waiting for the control plane availability...
2021-05-20 06:58:06 [✔]  saved kubeconfig as "/root/.kube/config"
2021-05-20 06:58:06 [ℹ]  no tasks
2021-05-20 06:58:06 [✔]  all EKS cluster resources for "eks-tidb3" have been created
2021-05-20 06:58:06 [ℹ]  adding identity "arn:aws:iam::xx:role/eksctl-eks-tidb3-nodegroup-tidb-1-NodeInstanceRole-xx" to auth ConfigMap
2021-05-20 06:58:06 [ℹ]  nodegroup "tidb-1d" has 0 node(s)
2021-05-20 06:58:06 [ℹ]  waiting for at least 1 node(s) to become ready in "tidb-1d"
Error: timed out (after 25m0s) waiting for at least 1 nodes to join the cluster and become ready in "tidb-1d"

[cPu1]

Thanks for opening a detailed issue. This may be related to #3719. We're already looking into it.

[yufan022]

May 20 06:57:30 ip-10-0-30-120 containerd: time="2021-05-20T06:57:30.807843287Z" level=info msg="Connect containerd service"
May 20 06:57:30 ip-10-0-30-120 containerd: time="2021-05-20T06:57:30.808304147Z" level=info msg="Get image filesystem path \"/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs\""
May 20 06:57:30 ip-10-0-30-120 containerd: time="2021-05-20T06:57:30.814444398Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
May 20 06:57:30 ip-10-0-30-120 containerd: time="2021-05-20T06:57:30.814494318Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1

@cPu1 Thanks for your quick reply. I found some error logs in the node, hoping to help. Be happy to provide more information if you need it

[yufan022]

@cPu1
Hi, May I ask how soon it will be released?
And I have another issue #3752 about fully-private cluster and OIDC. Could you please take a look at it? Thank you very much!