Skip to content

Commit

Permalink
Add examples, fix docker stop and add az-name var
Browse files Browse the repository at this point in the history
  • Loading branch information
jmthvt committed Jan 30, 2018
1 parent cc67497 commit d00de67
Showing 1 changed file with 59 additions and 9 deletions.
68 changes: 59 additions & 9 deletions docs/single-to-multi-master.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ Please follow all the backup steps before attempting it. Please read the
[etcd admin guide](https://github.com/coreos/etcd/blob/v2.2.1/Documentation/admin_guide.md)
before attempting it.

We can migrate from a single-master cluster to a multi-master cluster, but this is a complicated operation. It is easier to create a multi-master cluster using Kops (described [here](https://github.com/kubernetes/kops/blob/master/docs/high_availability.md)). If possible, try to plan this at time of cluster creation.

During this procedure, you will experience **downtime** on the API server, but
not on the end user services. During this downtime, existing pods will continue
to work, but you will not be able to create new pods and any existing pod that
Expand Down Expand Up @@ -65,6 +67,11 @@ a different AZ from the existing one.

```bash
$ kops create instancegroup master-<availability-zone2> --subnet <availability-zone2> --role Master
```
Example:

```bash
$ kops create ig master-eu-west-1b --subnet eu-west-1b --role Master
```

* ``maxSize`` and ``minSize`` should be 1,
Expand All @@ -77,6 +84,12 @@ also required. However, real EC2 instance is not required until the second maste

```bash
$ kops create instancegroup master-<availability-zone3> --subnet <availability-zone3> --role Master
```

Example:

```bash
$ kops create ig master-eu-west-1c --subnet eu-west-1c --role Master
```

* ``maxSize`` and ``minSize`` should be **0**,
Expand All @@ -91,25 +104,55 @@ reference a third one, even if we have not created it yet.*
$ kops edit cluster example.com
```

* In ``.spec.etcdClusters`` 2 new members in each cluster, one for each new
* In ``.spec.etcdClusters`` add 2 new members in each cluster, one for each new
availability zone.

```yaml
- instanceGroup: master-<availability-zone2>
name: <availability-zone2>
name: <availability-zone2-name>
- instanceGroup: master-<availability-zone3>
name: <availability-zone3>
name: <availability-zone3-name>
```
Example:
```yaml
etcdClusters:
- etcdMembers:
- instanceGroup: master-eu-west-1a
name: a
- instanceGroup: master-eu-west-1b
name: b
- instanceGroup: master-eu-west-1c
name: c
name: main
- etcdMembers:
- instanceGroup: master-eu-west-1a
name: a
- instanceGroup: master-eu-west-1b
name: b
- instanceGroup: master-eu-west-1c
name: c
name: events
```
## 3 - Add a new master
### a - Add a new member to the etcd clusters
**The clusters will stop to work until the new member is started**.
```bash
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone2> http://etcd-<availability-zone2>.internal.example.com:2380
$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone2> http://etcd-events-<availability-zone2>.internal.example.com:2381
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone2-name> http://etcd-<availability-zone2-name>.internal.example.com:2380 \
&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone2-name> http://etcd-events-<availability-zone2-name>.internal.example.com:2381
```

Example:

```bash
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-b http://etcd-b.internal.example.com:2380 \
&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-b http://etcd-events-b.internal.example.com:2381
```

### b - Launch the new master
Expand All @@ -132,7 +175,7 @@ Reinitialize the etcd instances:
* Delete the containers and the data directories:

```bash
root@ip-172-20-116-230:~# docker stop $(docker ps | grep "etcd:2.2.1" | awk '{print $1}')
root@ip-172-20-116-230:~# docker stop $(docker ps | grep "gcr.io/google_containers/etcd" | awk '{print $1}')
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/member/
root@ip-172-20-116-230:~# rm -r /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/member/
```
Expand Down Expand Up @@ -175,10 +218,17 @@ $ kops edit instancegroup master-<availability-zone3>
### b - Add a new member to the etcd clusters

```bash
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone3> http://etcd-<availability-zone3>.internal.example.com:2380
$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone3> http://etcd-events-<availability-zone3>.internal.example.com:2381
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone3-name> http://etcd-<availability-zone3-name>.internal.example.com:2380 \
&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone3-name> http://etcd-events-<availability-zone3-name>.internal.example.com:2381
```

Example:

```bash
$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-c http://etcd-c.internal.example.com:2380 \
&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-c http://etcd-events-c.internal.example.com:2381
```

### c - Launch the third master

```bash
Expand All @@ -197,7 +247,7 @@ $ kops edit instancegroup master-<availability-zone3>
* Delete the containers and the data directories:

```bash
root@ip-172-20-139-130:~# docker stop $(docker ps | grep "etcd:2.2.1" | awk '{print $1}')
root@ip-172-20-139-130:~# docker stop $(docker ps | grep "gcr.io/google_containers/etcd" | awk '{print $1}')
root@ip-172-20-139-130:~# rm -r /mnt/master-vol-019796c3511a91b4f//var/etcd/data-events/member/
root@ip-172-20-139-130:~# rm -r /mnt/master-vol-0c89fd6f6a256b686/var/etcd/data/member/
```
Expand Down

0 comments on commit d00de67

Please sign in to comment.