Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubefed v1.6 update #2899

Merged
merged 3 commits into from
Mar 21, 2017
Merged

kubefed v1.6 update #2899

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6c54d16
Update kubefed documentation.
madhusudancs Mar 17, 2017
5b50e8d
Address review comments.
madhusudancs Mar 18, 2017
b4ffa83
Address review comments.
madhusudancs Mar 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
272 changes: 253 additions & 19 deletions docs/tutorials/federation/set-up-cluster-federation-kubefed.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ existing federation control plane.
This guide explains how to administer a Kubernetes Cluster Federation
using `kubefed`.

> Note: `kubefed` is an alpha feature in Kubernetes 1.5.
> Note: `kubefed` is a beta feature in Kubernetes 1.6.

## Prerequisites

Expand All @@ -27,21 +27,37 @@ for installation instructions for your platform.

## Getting `kubefed`

Download the client tarball corresponding to Kubernetes version 1.5
or later
[from the release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md),
extract the binaries in the tarball to one of the directories
in your `$PATH` and set the executable permission on those binaries.
Download the client tarball corresponding to the latest release and
extract the binaries in the tarball with the commands:

Note: The URL in the curl command below downloads the binaries for
Linux amd64. If you are on a different platform, please use the URL
for the binaries appropriate for your platform. You can find the list
of available binaries on the [release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1).
```shell
# Linux
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz

# OS X
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-darwin-amd64.tar.gz
tar -xzvf kubernetes-client-darwin-amd64.tar.gz

# Windows
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s
https://storage.googleapis.com/kubernetes-release/release/stable.txt)/kubernetes-client-windows-amd64.tar.gz
tar -xzvf kubernetes-client-windows-amd64.tar.gz
```

> Note: The URLs in the curl commands above download the binaries for
`amd64`. If you are on a different architecture, please use a URL
appropriate for your architecture. You can find the list of available
binaries on the
[release page](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG.md#client-binaries-1).

Copy the extracted binaries to one of the directories in your `$PATH`
and set the executable permission on those binaries.


```shell
curl -O https://storage.googleapis.com/kubernetes-release/release/v1.5.2/kubernetes-client-linux-amd64.tar.gz
tar -xzvf kubernetes-client-linux-amd64.tar.gz
sudo cp kubernetes/client/bin/kubefed /usr/local/bin
sudo chmod +x /usr/local/bin/kubefed
sudo cp kubernetes/client/bin/kubectl /usr/local/bin
Expand Down Expand Up @@ -84,22 +100,32 @@ the following:

* Federation name
* `--host-cluster-context`, the `kubeconfig` context for the host cluster
* `--dns-provider`, one of `'google-clouddns'`, `aws-route53` or `coredns`
* `--dns-zone-name`, a domain name suffix for your federated services

If your host cluster is running in a non-cloud environment or an
environment that doesn't support common cloud primitives such as
load balancers, you might need additional flags. Please see the
[on-premises host clusters](#on-premises-host-clusters) section below.

The following example command deploys a federation control plane with
the name `fellowship`, a host cluster context `rivendell`, and the
domain suffix `example.com`:
domain suffix `example.com.`:

```shell
kubefed init fellowship --host-cluster-context=rivendell --dns-zone-name="example.com"
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="google-clouddns" \
--dns-zone-name="example.com."
```

The domain suffix specified in `--dns-zone-name` must be an existing
domain that you control, and that is programmable by your DNS provider.
It must also end with a trailing dot.

`kubefed init` sets up the federation control plane in the host
cluster and also adds an entry for the federation API server in your
local kubeconfig. Note that in the alpha release in Kubernetes 1.5,
local kubeconfig. Note that in the beta release in Kubernetes 1.6,
`kubefed init` does not automatically set the current context to the
newly deployed federation. You can set the current context manually by
running:
Expand All @@ -110,6 +136,190 @@ kubectl config use-context fellowship

where `fellowship` is the name of your federation.

### Basic and token authentication support

`kubefed init` by default only generates TLS certificates and keys
to authenticate with the federation API server and writes them to
your local kubeconfig file. If you wish to enable basic authentication
or token authentication for debugging purposes, you can enable them by
passing the `--apiserver-enable-basic-auth` flag or the
`--apiserver-enable-token-auth` flag.

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="google-clouddns" \
--dns-zone-name="example.com." \
--apiserver-enable-basic-auth=true \
--apiserver-enable-token-auth=true
```

### Passing command line arguments to federation components

`kubefed init` bootstraps a federation control plane with default
arguments to federation API server and federation controller manager.
Some of these arguments are derived from `kubefed init`'s flags.
However, you can override these command line arguments by passing
them via the appropriate override flags.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/them/a comma-separated list of arguments/

Or, if not here, can you make it clear somewhere other than the example that this should be a comma-separated list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"override these command line arguments by passing a comma-separated list of arguments" sounds a little awkward. It also probably redundant. So leaving this as is.


You can override the federation API server arguments by passing them
to `--apiserver-arg-overrides` and override the federation controller
manager arguments by passing them to
`--controllermanager-arg-overrides`.

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="google-clouddns" \
--dns-zone-name="example.com." \
--apiserver-arg-overrides="--anonymous-auth=false,--v=4" \
--controllermanager-arg-overrides="--controllers=services=false"
```

### Configuring a DNS provider

The Federated service controller programs a DNS provider to expose
federated services via DNS names. Certain cloud providers
automatically provide the configuration required to program the
DNS provider if the host cluster's cloud provider is same as the DNS
provider. In all other cases, you have to provide the DNS provider
configuration to your federation controller manager which will in-turn
be passed to the federated service controller. You can provide this
configuration to federation controller manager by storing it in a file
and passing the file's local filesystem path to `kubefed init`'s
`--dns-provider-config` flag. For example, save the config below in
`$HOME/coredns-provider.conf`.

```ini
[Global]
etcd-endpoints = http://etcd-cluster.ns:2379
zones = example.com.
```

And then pass this file to `kubefed init`:

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="coredns" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the sake of further clarity; do we also need to specify somewhere that dns-provider is a mandatory flag (or kubefed init wont work properly, if a known dns-provider is not specified) and then the dns-provider-conf becomes a mandatory option if the dns-provider (among the known providers) is coredns.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it is necessary to have all sorts of details about the various flags. Some of these belong to the reference documentation for the command.

Since --dns-provider is mandatory, the command won't even pass the validation in the beginning. So people are going to figure that out sooner than later, i.e. before modifying anything in the host cluster. So I want to leave that detail out.

Also, CoreDNS config is not mandatory. As in, it is not enforced, so I don't want to put that in the documentation here. It should be probably mentioned in the CoreDNS doc though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough!

--dns-zone-name="example.com." \
--dns-provider-config="$HOME/coredns-provider.conf"
```

### On-premises host clusters

#### API server service type

`kubefed init` exposes the federation API server as a Kubernetes
[service](/docs/user-guide/services/) on the host cluster. By default,
this service is exposed as a
[load balanced service](/docs/user-guide/services/#type-loadbalancer).
Most on-premises and bare-metal enviroments, and some cloud
environments lack support for load balanced services. `kubefed init`
allows exposing the federation API server as a
[`NodePort` service](/docs/user-guide/services/#type-nodeport) on
such environments. This can be accomplished by passing
the `--api-server-service-type=NodePort` flag. You can also specify
the preferred address to advertise the federation API server by
passing the `--api-server-advertise-address=<IP-address>`
flag. Otherwise, one of the host cluster's node address is chosen as
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we also need to say, that one of the external IP's will be chosen and/or a catch, if a publicIP or a legacyIP is not available, the auto assignment of the IP will remain empty.

the default.

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="google-clouddns" \
--dns-zone-name="example.com." \
--api-server-service-type="NodePort" \
--api-server-advertise-address="10.0.10.20"
```

#### Provisioning storage for etcd

Federation control plane stores its state in
[`etcd`](https://coreos.com/etcd/docs/latest/).
[`etcd`](https://coreos.com/etcd/docs/latest/) data must be stored in
a persistent storage volume to ensure correct operation across
federation control plane restarts. On host clusters that support
[dynamic provisioning of storage volumes](/docs/user-guide/persistent-volumes/#dynamic),
`kubefed init` dynamically provisions a
[`PersistentVolume`](/docs/user-guide/persistent-volumes/#persistent-volumes)
and binds it to a
[`PersistentVolumeClaim`](/docs/user-guide/persistent-volumes/#persistentvolumeclaims)
to store [`etcd`](https://coreos.com/etcd/docs/latest/) data. If your
host cluster doesn't support dynamic provisioning, you can also
statically provision a
[`PersistentVolume`](/docs/user-guide/persistent-volumes/#persistent-volumes).
`kubefed init` creates a
[`PersistentVolumeClaim`](/docs/user-guide/persistent-volumes/#persistentvolumeclaims)
that has the following configuration:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.alpha.kubernetes.io/storage-class: "yes"
labels:
app: federated-cluster
name: fellowship-federation-apiserver-etcd-claim
namespace: federation-system
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
```

To statically provision a
[`PersistentVolume`](/docs/user-guide/persistent-volumes/#persistent-volumes),
you must ensure that the
[`PersistentVolume`](/docs/user-guide/persistent-volumes/#persistent-volumes)
that you create has the matching storage class, access mode and
at least as much capacity as the requested
[`PersistentVolumeClaim`](/docs/user-guide/persistent-volumes/#persistentvolumeclaims).

Alternatively, you can disable persistent storage completely
by passing `--etcd-persistent-storage=false` to `kubefed init`.
However, we do not recommended this because your federation control
plane cannot survive restarts in this mode.

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="google-clouddns" \
--dns-zone-name="example.com." \
--etcd-persistent-storage=false
```

`kubefed init` still doesn't support attaching an existing
[`PersistentVolumeClaim`](/docs/user-guide/persistent-volumes/#persistentvolumeclaims)
to the federation control plane that it bootstraps. We are planning to
support this in a future version of `kubefed`.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madhusudancs, in L235-L238, the documentation is slightly confusing. when --etcd-persistent-storage=false we disable persistent storage (either dynamic or statically provisioned pv will be disabled).

And for L248-L251, If user creates statically provisioned pv with exact same requirements of pvc, the current mechanism already supports binding to statically provisioned pv. only that the pvc configuration may not be know to user.

Copy link
Contributor Author

@madhusudancs madhusudancs Mar 18, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madhusudancs, in L235-L238, the documentation is slightly confusing. when --etcd-persistent-storage=false we disable persistent storage (either dynamic or statically provisioned pv will be disabled).

Ok, first of all I misunderstood this. Thanks for clarifying.

The reason why I misunderstood this was because we use the annotation "volume.alpha.kubernetes.io/storage-class": "yes" in our PVC and I thought it wasn't possible to create a PV manually with that storage class. So I thought one has to pass --etcd-persistent-storage=false and create a PV and a PVC themselves. Isn't that correct?

And for L248-L251, If user creates statically provisioned pv with exact same requirements of pvc, the current mechanism already supports binding to statically provisioned pv. only that the pvc configuration may not be know to user.

If it possible to just create a PV that our PVC can bind to, then we can document our PVC's storage class and access modes here so that people can create a PV for that configuration. Then we can ask people to set --etcd-persistent-storage=false only when they don't need persistence.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if --etcd-persistent-storage=false PVC is not created by kubefed, thereby disabling persistent storage altogether.

if user creates required PV statically beforehand matching the requirements of PVC created by kubefed, then PVC is able to bind to the Statically created PV. So definitely we should be documenting the PVC's storage class, access modes and size. This will enable users to create PV beforehand, if necessary.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understood this after your previous comment. But my question is, is it possible to create a PV manually with "volume.alpha.kubernetes.io/storage-class": "yes"?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure of this @madhusudancs. In my local setup i had created a PV without this storage-class statically and i was able to bind to the PVC created by kubefed.
So my guess is this storage-class applies to dynamic provisioning of PV, something meaningful for cloud-provider. Just a guess.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shashidharatd Oh I see. That's useful information. Thanks!

Documented the PVC configuration along with a description. PTAL.

#### CoreDNS support

Federated services now support [CoreDNS](https://coredns.io/) as one
of the DNS providers. If you are running your clusters and federation
in an environment that does not have access to cloud-based DNS
providers, then you can run your own [CoreDNS](https://coredns.io/)
instance and publish the federated service DNS names to that server.

You can configure your federation to use
[CoreDNS](https://coredns.io/), by passing appropriate values to
`kubefed init`'s `--dns-provider` and `--dns-provider-config` flags.

```shell
kubefed init fellowship \
--host-cluster-context=rivendell \
--dns-provider="coredns" \
--dns-zone-name="example.com." \
--dns-provider-config="$HOME/coredns-provider.conf"
```

For more information see
[Setting up CoreDNS as DNS provider for Cluster Federation](/docs/tutorials/federation/set-up-coredns-provider-federation/)

## Adding a cluster to a federation

Expand All @@ -121,8 +331,19 @@ To use `kubefed join`, you'll need to provide the name of the cluster
you want to add to the federation, and the `--host-cluster-context`
for the federation control plane's host cluster.

> Note: The name that you provide to the `join` command is used as the
joining cluster's identity in federation. This name should adhere to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a naming rules section below:
Does it make more sense to list all this detail there, and a one liner note here something like:
Note: The name that you provide to the `join` command is used as the joining cluster's identity in federation. This name needs to adhere to specific rules, some details of which are listed in naming rules section below:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note is in response to the direct feedback I got from different people (issue #2353). It is not clear why you need to name a cluster while joining it and all the aliasing that happens between context names and cluster names in federation. That's why a note here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, Please leave it as it is now!

the rules described in the
[identifiers doc](/docs/user-guide/identifiers/#names). If the context
corresponding to your joining cluster conforms to these rules then you
can use the same name in the join command. Otherwise, you will have to
choose a different name for your cluster's identity. For more
information, please see the
[naming rules and customization](#naming-rules-and-customization)
section below.

The following example command adds the cluster `gondor` to the
federation with host cluster `rivendell`:
federation running on host cluster `rivendell`:

```
kubefed join gondor --host-cluster-context=rivendell
Expand All @@ -136,8 +357,9 @@ cluster.

### Naming rules and customization

The cluster name you supply to `kubefed join` must be a valid RFC 1035
label.
The cluster name you supply to `kubefed join` must be a valid
[RFC 1035](https://www.ietf.org/rfc/rfc1035.txt) label and are
enumerated in the [Identifiers doc](/docs/user-guide/identifiers/#names).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not quite sure I follow what you're saying here. Is it that the identifier must be a valid RFC 1035 label and adhere to the requirements enumerated in the Identifiers page?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. And they are the same. Identifiers doc enumerates RFC 1035 rules.

Any suggestions for improving this or to make this sound better?


Furthermore, federation control plane requires credentials of the
joined clusters to operate on them. These credentials are obtained
Expand Down Expand Up @@ -182,6 +404,18 @@ specification, all you need to do is supply the secret name via the
`--secret-name` flag. `kubefed join` automatically creates the secret
for you.

### `kube-dns` configuration

`kube-dns` configuration must be updated in each joining cluster to
enable federated service discovery. If the joining Kubernetes cluster
is version 1.5 or newer and your `kubefed` is version 1.6 or newer,
then this configuration is automatically managed for you when the
clusters are joined or unjoined using `kubefed` `join` or `unjoin`
commands.

In all other cases, you must update `kube-dns` configuration manually
as described in the
[Updating KubeDNS section of the admin guide](/docs/admin/federation/)

## Removing a cluster from a federation

Expand All @@ -197,7 +431,7 @@ kubefed unjoin gondor --host-cluster-context=rivendell
## Turning down the federation control plane:

Proper cleanup of federation control plane is not fully implemented in
this alpha release of `kubefed`. However, for the time being, deleting
this beta release of `kubefed`. However, for the time being, deleting
the federation system namespace should remove all the resources except
the persistent storage volume dynamically provisioned for the
federation control plane's etcd. You can delete the federation
Expand Down