kubeadm join labels node as master

[joshuacox]

What keywords did you search in kubeadm issues before filing this one?

node label kubeadm join

this one is similar:
#202

and I've been subscribed to that for a very long time, but this is unique to the external etcd method

Choose one: BUG REPORT or FEATURE REQUEST

/kind bug

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:51:33Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

kubectl version --kubeconfig=/etc/kubernetes/admin.conf
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

ubuntu xenial on baremetal

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux testymaster1 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

While following the external etcd method

my nodes were labeled as master upon join:

NAME           STATUS    ROLES         AGE       VERSION
testymaster1   Ready     master        20m       v1.12.2
testymaster2   Ready     master        19m       v1.12.2
testymaster3   Ready     master        19m       v1.12.2
testynode1     Ready     master,node   19m       v1.12.2
testynode2     Ready     master,node   19m       v1.12.2
testynode3     Ready     master,node   19m       v1.12.2

What you expected to happen?

I expected the same thing to happen when I do the stacked method

NAME              STATUS    ROLES     AGE       VERSION
stackedingress1   Ready     node      35m       v1.12.2
stackedmaster1    Ready     master    41m       v1.12.2
stackedmaster2    Ready     master    40m       v1.12.2
stackedmaster3    Ready     master    37m       v1.12.2
stackednode1      Ready     node      35m       v1.12.2
stackednode2      Ready     node      35m       v1.12.2
stackednode3      Ready     node      35m       v1.12.2

How to reproduce it (as minimally and precisely as possible)?

Follow the external etcd method and then run the resulting kubeadm join command.

Anything else we need to know?

I had my fair share of issues with this method which can be seen here. Of which I ended up with two scripts that I am using to initialize my cluster:

prep the etcd nodes by running this script on the primary etcd node:

https://gist.github.com/joshuacox/9df2a029b04e63443b62c2824cf5fb95

 tar zcf - scripts/etcd-test.sh|  ssh [email protected] 'tar zxvf -;cd scripts; bash etcd-test.sh 10.0.23.218 10.0.23.219 10.0.23.220'; 

and then initialize a master, this script can be ran on any host that has been keyed for ssh access to both the master and the primary etcd node

https://gist.github.com/joshuacox/f0f0b25e51df5638f3778d80d4af8c63

bash scripts/final_master.sh 10.0.23.215 10.0.23.218

and the join command looks like this:

kubeadm join 10.0.23.108:6443 --token this_token --discovery-token-ca-cert-hash sha256:this_shah --experimental-control-plane

[joshuacox]

as an aside note, the apiserver fails with this:

F1122 14:36:34.060030       1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry [https://10.0.23.111:2379 https://10.0.23.121:2379 https://10.0.23.125:2379] /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/etcd/ca.crt true true 1000 0xc420177e60 <nil> 5m0s 1m0s}), err (open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory)

which is most likely not required once I get rid of the master label on the node, however, I did have to copy a number of certs before kubeadm successfully completed its join. I've got a WIP script which combines it all here:

https://github.com/joshuacox/kubash/blob/1.12.2/scripts/final_node.sh

[joshuacox]

[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Marking the node extetcdnode1 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node extetcdnode1 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The kubernetes control plane instances scaled up.

To start administering your cluster from this node, you need to run the following as a regular user:

        mkdir -p $HOME/.kube
        sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
        sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

[joshuacox]

ok one note, do not add --experimental-control-plane to the node join but do add:

--ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt

and copy all the certs as I have done in those scripts, then run the joins and afterwards label the node as node.

[luxas]

@fabriziopandini bugfix needed for v1.13?

[joshuacox]

I think its more of a doc fix. Or get kubeadm join to ignore the ca.crt existing beforehand.

[fabriziopandini]

@luxas
I'm just finished tests on v1.13.0-beta.2 without noticing nodes with wrong labels. I will double check

@joshuacox

ok one note, do not add --experimental-control-plane to the node join but do add: --ignore-preflight-errors=FileAvailable--etc-kubernetes-pki-ca.crt

When joining a secondary control-plane you should add --experimental-control-plane (and ca.crt check will be skipped automatically)
When joining a worker node you should omit --experimental-control-plane and ca.crt should not exist

[joshuacox]

@fabianofranz you are indeed correct, I have modified as per your recommendations:

https://gist.github.com/joshuacox/95aad9bee0c7e49e735ec3ec553b24ca

and all seems to work, back to the drawing board on a few things to find out where I went wrong! Closing.

[fabriziopandini]

@joshuacox happy to hear it worked!
Thanks for the feedback!