Planning and testing upgrade Live/Manager cluster to Kubernetes 1.20.x

[jasonBirchall]

Background

We want to test our process for upgrading on EKS.

The purpose of this ticket is to plan the upgrade and identify any issues that might result in users changes. We don't want to proceed with the upgrade if users are required to make changes in support of the upgrade due to ongoing migration work.

Questions / Assumptions

Are there any changes in the upgrade that require users to make any changes?

Definition of done

• we have confirmed there are no user impacting changes
• we can decide on whether to proceed with the upgrade

Reference

How to write good user stories

[davidread]

Break this ticket down into smaller tickets:

• Check if there are any hard deprecations in 1.20 to impose on users - that might be a deal breaker for doing this upgrade at the same time of the EKS migration
• Improve the runbook for zero-downtime EKS upgrades, comparing with all the steps we did in the live-1/kops upgrade runbook. Test it in a test cluster.
• Upgrade live
• Upgrade manager

[razvan-moj]

Checked https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#changelog-since-v1190, looks fine

[razvan-moj]

Created a 1.19 EKS test cluster, upgraded via web interface, looks fine

[jasonBirchall]

There's a fairly decent tool that you can run against your cluster that gives you a report of changed API's:
(run against a test cluster on 1.19)

https://github.com/devtron-labs/silver-surfer

➜ ./kubedd --target-kubernetes-version 1.20    

Results for cluster at version 1.19 to 1.20
-------------------------------------------
>>>> Deprecated API Version's <<<<
 Namespace   Name                              Kind                             API Version (Current Available)        Replace With API Version (Latest Available)   Migration Status                                   
 undefined   vpc-resource-validating-webhook   ValidatingWebhookConfiguration   admissionregistration.k8s.io/v1beta1   admissionregistration.k8s.io/v1               1 issue(s): fix issues before migration  
 undefined   pod-identity-webhook              MutatingWebhookConfiguration     admissionregistration.k8s.io/v1beta1   admissionregistration.k8s.io/v1               1 issue(s): fix issues before migration  
 undefined   vpc-resource-mutating-webhook     MutatingWebhookConfiguration     admissionregistration.k8s.io/v1beta1   admissionregistration.k8s.io/v1               1 issue(s): fix issues before migration  

>>> Validation Errors against latest api version, should be resolved before migration <<<
 Namespace   Name                              Kind                             API Version (Latest Available)    Field                                Reason                                         
 undefined   vpc-resource-validating-webhook   ValidatingWebhookConfiguration   admissionregistration.k8s.io/v1   webhooks/0/admissionReviewVersions   property "admissionReviewVersions" is missing  
 undefined   pod-identity-webhook              MutatingWebhookConfiguration     admissionregistration.k8s.io/v1   webhooks/0/admissionReviewVersions   property "admissionReviewVersions" is missing  
 undefined   vpc-resource-mutating-webhook     MutatingWebhookConfiguration     admissionregistration.k8s.io/v1   webhooks/0/admissionReviewVersions   property "admissionReviewVersions" is missing  

[jasonBirchall]

The runbook appears to be missing key upgrades, like coredns, kube-proxy and kubectl updates.

[razvan-moj]

We're blocked by several bugs / missing features:

• CNI needs a serviceaccount otherwise we grant nodes a privilege AWS does not recommend, but when trying to use data "kubernetes_service_account" in https://github.com/ministryofjustice/cloud-platform-terraform-irsa to edit the aws-node sa, it fails because Data resource for Service Account fails to find default token hashicorp/terraform-provider-kubernetes#1104

• the EKS provider doesn't have support for add-ons, but work is in progress (feat: Add support for addons terraform-aws-modules/terraform-aws-eks#1443) and us coming up with a workaround might be wasted effort

• installing the CNI via Helm chart (https://github.com/ministryofjustice/cloud-platform-terraform-vpc-cni) needs a null_resource exec that is bound to bit us later

[razvan-moj]

• kubectl & other deps updated in https://github.com/ministryofjustice/cloud-platform-tools-image/pull/87/files

• kubernetes provider pinning needs to be removed, see https://github.com/ministryofjustice/cloud-platform-infrastructure/pull/1426/files

• https://github.com/ministryofjustice/cloud-platform-infrastructure/blob/main/terraform/aws-accounts/cloud-platform-aws/vpc/eks/cluster.tf#L107 needs to become a parameter that allows us to have clusters running different versions

[razvan-moj]

• tools image needs changing in all references eg https://github.com/ministryofjustice/cloud-platform-terraform-concourse/blob/main/pipelines/manager/main/environments-live.yaml#L30

[vijay-veeranki]

There is a lot more info from various developers facing problems with addons(coredns, kube-proxy and VPC CNI ) in this issue

Good Read about addons:
https://itnext.io/amazon-eks-add-ons-implemented-with-terraform-66a49fad4174

The conclusion from above:

In general, if you are trying to make any changes to coredns or kube-proxy or even VPC CNI then don’t use EKS add-ons at this early stage. You simply won’t be able to customise these Kubernetes operators/applications.

I have to say that from the start of the deployment of the first add-on, there were issues. Like with many AWS products at launch, I don’t believe this is production-ready. However, it works well if you fix all the issues and don’t have to make any customisations. That said, you do require good Kubernetes RBAC knowledge to spot and quickly fix permission issues.

[vijay-veeranki]

Terraform code to manage addons, but configuring is not available yet.

resource "aws_eks_addon" "vpc_cni" {
  count = var.addon_create_vpc_cni ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "vpc-cni"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_vpc_cni_version

  tags = local.tags
}

resource "aws_eks_addon" "kube_proxy" {
  count = var.addon_create_kube_proxy ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "kube-proxy"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_kube_proxy_version

  tags = local.tags
}

resource "aws_eks_addon" "coredns" {
  count = var.addon_create_coredns ? 1 : 0

  cluster_name      = module.eks.cluster_id
  addon_name        = "coredns"
  resolve_conflicts = "OVERWRITE"
  addon_version     = var.addon_coredns_version

  tags = local.tags
}

[davidread]

Raz says this is done in other stories