single-to-multi-master doc: Create all master instances groups at first

[dtan4]

Why

Current single-to-multi-master says:

• Create the third master instance group after establishing 2-masters cluster
• Add second and third master instance groups to .spec.etcdClusters, even if we have not created it yet.

I tried to single-to-multi master migration with the above document, but I got an the below error:

$ kops update cluster
Using cluster from kubectl context: k8s.test-cluster.wantedlyapp.com


*********************************************************************************

A new kubernetes version is available: 1.5.7
Upgrading is recommended (try kops upgrade cluster)

More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.5.7

*********************************************************************************


error building tasks: InstanceGroup not found (for etcd c2/main): "master-ap-northeast-1c-2"

kops update cluster does not accept nonexist instance group...

After creating the third master instance group with empty size, it worked well and I could finish migration successfully.

What

Revise document to create all master instance groups at first.

---

This change is 

[k8s-ci-robot]

Hi @dtan4. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with @k8s-bot ok to test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[chrislovecnm]

@k8s-bot ok to test

[chrislovecnm]

@MrTrustor can you comment on this? I have not done this before, so approving changes to the doc would need me to go through the documentation

[MrTrustor]

I have not tested this procedure since writing this doc, so the behavior of Kops may have changed since then. The proposed changes seem reasonable enough.
@chrislovecnm Do you need me to actually test the new procedure?

[chrislovecnm]

@MrTrustor that would be amazing if you could, no worries if you cannot. @dtan4 can you outline how you have tested this process?

[dtan4]

I've upgraded current cluster with the steps below:

1. create single-master cluster

cluster name: cluster.example.com

• master: ap-northeast-1c
• nodes: ap-northeast-1b, ap-northeast-1c

2. create instance groups

(ap-northeast-1 users can use 2 AZs only...)

kops create ig master-ap-northeast-1b --subnet ap-northeast-1b --role Master

# change maxSize, minSize to 0
kops create ig master-ap-northeast-1c-2 --subnet ap-northeast-1c --role Master

3. add second master

a. edit cluster

kops edit cluster

add this to both etcdMembers:

    - instanceGroup: master-ap-northeast-1b
      name: b
    - instanceGroup: master-ap-northeast-1c-2
      name: c2

b. add etcd member

etcd Pod:

$ kubectl get po --namespace kube-system | grep etcd
etcd-server-events-ip-10-1-72-52.ap-northeast-1.compute.internal        1/1       Running   0          70d
etcd-server-ip-10-1-72-52.ap-northeast-1.compute.internal               1/1       Running   0          70d

kubectl --namespace=kube-system exec etcd-server-ip-10-1-72-52.ap-northeast-1.compute.internal -- etcdctl member add etcd-b http://etcd-b.internal.cluster.example.com:2380
kubectl --namespace=kube-system exec etcd-server-events-ip-10-1-72-52.ap-northeast-1.compute.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-b http://etcd-events-b.internal.cluster.example.com:2381

c. update cluster

kops update cluster --yes

d. update etcd in new master

ssh -i ~/.ssh/key.rsa admin@<new master>

sudo -i
systemctl stop kubelet
systemctl stop protokube
sudo vim /etc/kubernetes/manifests/etcd-events.manifest
sudo vim /etc/kubernetes/manifests/etcd.manifest
docker stop $(docker ps | grep "etcd:2.2.1" | awk '{print $1}')
rm -r /mnt/<id>/var/etcd/data-events/member/
rm -r /mnt/<id>/var/etcd/data/member/
systemctl start kubelet
exit
exit

4. add third master

a. increase ASG size of third-master instance group

# increase minSize and maxSize to 1
kops edit ig master-ap-northeast-1c-2

b. add etcd member

kubectl --namespace=kube-system exec etcd-server-ip-10-1-89-92.ap-northeast-1.compute.internal -- etcdctl member add etcd-c2 http://etcd-c2.internal.cluster.example.com:2380
kubectl --namespace=kube-system exec etcd-server-events-ip-10-1-89-92.ap-northeast-1.compute.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-c2 http://etcd-events-c2.internal.cluster.example.com:2381

c. update cluster

kops update cluster --yes

d. update etcd in new master

ssh -i ~/.ssh/key.rsa admin@<new master>

sudo -i
systemctl stop kubelet
systemctl stop protokube
sudo vim /etc/kubernetes/manifests/etcd-events.manifest
sudo vim /etc/kubernetes/manifests/etcd.manifest
docker stop $(docker ps | grep "etcd:2.2.1" | awk '{print $1}')
rm -r /mnt/<id>/var/etcd/data-events/member/
rm -r /mnt/<id>/var/etcd/data/member/
systemctl start kubelet
exit
exit

[chrislovecnm]

Thanks, @dtan4, let's get this in an iterate

[chrislovecnm]

Thanks