From 75ed07f7169f1b5e96e0684c86e6f4699eefd779 Mon Sep 17 00:00:00 2001 From: Jordan Liggitt Date: Fri, 7 Dec 2018 19:06:31 -0500 Subject: [PATCH] Add information on supported version skew and upgrade order (#11060) * Add information on supported version skew and upgrade order * address comments * added cloud-controller-manager * format headers --- .../reference/using-api/deprecation-policy.md | 2 +- .../independent/create-cluster-kubeadm.md | 6 +- .../docs/setup/independent/install-kubeadm.md | 6 +- content/en/docs/setup/version-skew-policy.md | 148 ++++++++++++++++++ .../users/cluster-operator/foundational.md | 3 +- data/setup.yml | 2 + 6 files changed, 161 insertions(+), 6 deletions(-) create mode 100644 content/en/docs/setup/version-skew-policy.md diff --git a/content/en/docs/reference/using-api/deprecation-policy.md b/content/en/docs/reference/using-api/deprecation-policy.md index a6a8f190cd7bb..90364440e15e9 100644 --- a/content/en/docs/reference/using-api/deprecation-policy.md +++ b/content/en/docs/reference/using-api/deprecation-policy.md @@ -87,7 +87,7 @@ no less than:** * **Beta: 9 months or 3 releases (whichever is longer)** * **Alpha: 0 releases** -This covers the maximum supported version skew of 2 releases. +This covers the [maximum supported version skew of 2 releases](/docs/setup/version-skew-policy/). {{< note >}} Until [#52185](https://github.com/kubernetes/kubernetes/issues/52185) is diff --git a/content/en/docs/setup/independent/create-cluster-kubeadm.md b/content/en/docs/setup/independent/create-cluster-kubeadm.md index c69ec051326a1..81b38975765be 100644 --- a/content/en/docs/setup/independent/create-cluster-kubeadm.md +++ b/content/en/docs/setup/independent/create-cluster-kubeadm.md @@ -589,8 +589,10 @@ Due to that we can't see into the future, kubeadm CLI vX.Y may or may not be abl Example: kubeadm v1.8 can deploy both v1.7 and v1.8 clusters and upgrade v1.7 kubeadm-created clusters to v1.8. -Please also check our [installation guide](/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl) -for more information on the version skew between kubelets and the control plane. +These resources provide more information on supported version skew between kubelets and the control plane, and other Kubernetes components: + +* Kubernetes [version and version-skew policy](/docs/setup/version-skew-policy/) +* Kubeadm-specific [installation guide](/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl) ## kubeadm works on multiple platforms {#multi-platform} diff --git a/content/en/docs/setup/independent/install-kubeadm.md b/content/en/docs/setup/independent/install-kubeadm.md index cbf1c8ebb483e..317ffed7d14a9 100644 --- a/content/en/docs/setup/independent/install-kubeadm.md +++ b/content/en/docs/setup/independent/install-kubeadm.md @@ -119,8 +119,10 @@ This is because kubeadm and Kubernetes require [special attention to upgrade](/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/). {{}} -For more information on version skews, please read our -[version skew policy](/docs/setup/independent/create-cluster-kubeadm/#version-skew-policy). +For more information on version skews, see: + +* Kubernetes [version and version-skew policy](/docs/setup/version-skew-policy/) +* Kubeadm-specific [version skew policy](/docs/setup/independent/create-cluster-kubeadm/#version-skew-policy) {{< tabs name="k8s_install" >}} {{% tab name="Ubuntu, Debian or HypriotOS" %}} diff --git a/content/en/docs/setup/version-skew-policy.md b/content/en/docs/setup/version-skew-policy.md new file mode 100644 index 0000000000000..6903c6cf5b4c8 --- /dev/null +++ b/content/en/docs/setup/version-skew-policy.md @@ -0,0 +1,148 @@ +--- +reviewers: +- sig-api-machinery +- sig-architecture +- sig-cli +- sig-cluster-lifecycle +- sig-node +- sig-release +title: Kubernetes Version and Version Skew Support Policy +content_template: templates/concept +weight: 70 +--- + +{{% capture overview %}} +This document describes the maximum version skew supported between various Kubernetes components. +Specific cluster deployment tools may place additional restrictions on version skew. +{{% /capture %}} + +{{% capture body %}} + +## Supported versions + +Kubernetes versions are expressed as **x.y.z**, +where **x** is the major version, **y** is the minor version, and **z** is the patch version, following [Semantic Versioning](http://semver.org/) terminology. +For more information, see [Kubernetes Release Versioning](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/release/versioning.md#kubernetes-release-versioning). + +The Kubernetes project maintains release branches for the most recent three minor releases. + +Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility. +Patch releases are cut from those branches at a regular cadence, or as needed. +This decision is owned by the [patch release manager](https://github.com/kubernetes/sig-release/blob/master/release-team/role-handbooks/patch-release-manager/README.md#release-timing). +The patch release manager is a member of the [release team for each release](https://github.com/kubernetes/sig-release/tree/master/releases/). + +Minor releases occur approximately every 3 months, so each minor release branch is maintained for approximately 9 months. + +## Supported version skew + +### kube-apiserver + +In [highly-availabile (HA) clusters](https://kubernetes.io/docs/setup/independent/high-availability/), the newest and oldest `kube-apiserver` instances must be within one minor version. + +Example: + +* newest `kube-apiserver` is at **1.13** +* other `kube-apiserver` instances are supported at **1.13** and **1.12** + +### kubelet + +`kubelet` must not be newer than `kube-apiserver`, and may be up to two minor versions older. + +Example: + +* `kube-apiserver` is at **1.13** +* `kubelet` is supported at **1.13**, **1.12**, and **1.11** + +{{< note >}} +If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the allowed `kubelet` versions. +{{}} + +Example: + +* `kube-apiserver` instances are at **1.13** and **1.12** +* `kubelet` is supported at **1.12**, and **1.11** (**1.13** is not supported because that would be newer than the `kube-apiserver` instance at version **1.12**) + +### kube-controller-manager, kube-scheduler, and cloud-controller-manager + +`kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` must not be newer than the `kube-apiserver` instances they communicate with. They are expected to match the `kube-apiserver` minor version, but may be up to one minor version older (to allow live upgrades). + +Example: + +* `kube-apiserver` is at **1.13** +* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **1.13** and **1.12** + +{{< note >}} +If version skew exists between `kube-apiserver` instances in an HA cluster, and these components can communicate with any `kube-apiserver` instance in the cluster (for example, via a load balancer), this narrows the allowed versions of these components. +{{< /note >}} + +Example: + +* `kube-apiserver` instances are at **1.13** and **1.12** +* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` communicate with a load balancer that can route to any `kube-apiserver` instance +* `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` are supported at **1.12** (**1.13** is not supported because that would be newer than the `kube-apiserver` instance at version **1.12**) + +### kubectl + +`kubectl` is supported within one minor version (older or newer) of `kube-apiserver`. + +Example: + +* `kube-apiserver` is at **1.13** +* `kubectl` is supported at **1.14**, **1.13**, and **1.12** + +{{< note >}} +If version skew exists between `kube-apiserver` instances in an HA cluster, this narrows the supported `kubectl` versions. +{{< /note >}} + +Example: + +* `kube-apiserver` instances are at **1.13** and **1.12** +* `kubectl` is supported at **1.13** and **1.12** (other versions would be more than one minor version skewed from one of the `kube-apiserver` components) + +## Supported component upgrade order + +The supported version skew between components has implications on the order in which components must be upgraded. +This section describes the order in which components must be upgraded to transition an existing cluster from version **1.n** to version **1.(n+1)**. + +### kube-apiserver + +Pre-requisites: + +* In a single-instance cluster, the existing `kube-apiserver` instance is **1.n** +* In an HA cluster, all `kube-apiserver` instances are at **1.n** or **1.(n+1)** (this ensures maximum skew of 1 minor version between the oldest and newest `kube-apiserver` instance) +* The `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` instances that communicate with this server are at version **1.n** (this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version) +* `kubelet` instances on all nodes are at version **1.n** or **1.(n-1)** (this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version) +* Registered admission webhooks are able to handle the data the new `kube-apiserver` instance will send them: + * `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects are updated to include any new versions of REST resources added in **1.(n+1)** + * The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in **1.(n+1)** + +Upgrade `kube-apiserver` to **1.(n+1)** + +{{< note >}} +Project policies for [API deprecation](https://kubernetes.io/docs/reference/using-api/deprecation-policy/) and +[API change guidelines](https://github.com/kubernetes/community/blob/master/contributors/devel/api_changes.md) +require `kube-apiserver` to not skip minor versions when upgrading, even in single-instance clusters. +{{< /note >}} + +### kube-controller-manager, kube-scheduler, and cloud-controller-manager + +Pre-requisites: + +* The `kube-apiserver` instances these components communicate with are at **1.(n+1)** (in HA clusters in which these control plane components can communicate with any `kube-apiserver` instance in the cluster, all `kube-apiserver` instances must be upgraded before upgrading these components) + +Upgrade `kube-controller-manager`, `kube-scheduler`, and `cloud-controller-manager` to **1.(n+1)** + +### kubelet + +Pre-requisites: + +* The `kube-apiserver` instances the `kubelet` communicates with are at **1.(n+1)** + +Optionally upgrade `kubelet` instances to **1.(n+1)** (or they can be left at **1.n** or **1.(n-1)**) + +{{< warning >}} +Running a cluster with `kubelet` instances that are persistently two minor versions behind `kube-apiserver` is not recommended: + +* they must be upgraded within one minor version of `kube-apiserver` before the control plane can be upgraded +* it increases the likelihood of running `kubelet` versions older than the three maintained minor releases +{{}} diff --git a/content/en/docs/user-journeys/users/cluster-operator/foundational.md b/content/en/docs/user-journeys/users/cluster-operator/foundational.md index d943a0e0cca9f..888d8b47f88b1 100644 --- a/content/en/docs/user-journeys/users/cluster-operator/foundational.md +++ b/content/en/docs/user-journeys/users/cluster-operator/foundational.md @@ -65,7 +65,8 @@ These resources are covered in a number of articles within the Kubernetes docume As a cluster operator you may not need to use all these resources, although you should be familiar with them to understand how the cluster is being used. There are a number of additional resources that you should be aware of, some listed under [Intermediate Resources](/docs/user-journeys/users/cluster-operator/intermediate#section-1). -You should also be familiar with [how to manage kubernetes resources](/docs/concepts/cluster-administration/manage-deployment/). +You should also be familiar with [how to manage kubernetes resources](/docs/concepts/cluster-administration/manage-deployment/) +and [supported versions and version skew between cluster components](/docs/setup/version-skew-policy/). ## Get information about your cluster diff --git a/data/setup.yml b/data/setup.yml index f7cd2ffee014b..8510c915b07e0 100644 --- a/data/setup.yml +++ b/data/setup.yml @@ -135,3 +135,5 @@ toc: - title: Building High-Availability Clusters path: /docs/admin/high-availability/building/ + +- docs/setup/version-skew-policy.md