Release Proposal v2.27

[tico88612]

Urgent Upgrade Notes

(No, really, you MUST read this before you upgrade)

• Action required
  Change kubeadm_patches format to use an array of inline patch instead of patch files.
  See the example for new format. (Use in inventory variables rather than patch files for kubeadm_patches #11521, @VannTen)
• Action required
  Removes the generation of static tokens for every node in the cluster when kube_token_auth: true (Remove generation of static tokens for cluster members #11567, @VannTen)
• Action required
  The kubelet_node_{config_extra_args,custom_flags} are removed. Use kubelet_{config_extra_args,custom_flags} in <your_inventory>/group_vars/kube_node.yml.
  The {kube,system}_master_{cpu,memory,ephemeral-storage,pid} are removed. Use the {kube,system}_{cpu,memory,ephemeral-storage,pid} variables in <your_inventory>/group_vars/kube_control_plane.yml. kubelet_custom_flags` can no longer be a string, an array is required. (Refactor kubernetes/node templates #10643, @VannTen)
• Action required
  k8s_cluster group is now automatically defined, it can be removed from your inventory if you're not using it for group_vars (Only require minimum structure in inventory, compute the rest #11559, @VannTen)
• Action required
  kubeadm_ignore_preflight_errors is introduced to ignore specific preflight checks from kubeadm. The previous was effectively all, so some errors might surface during upgrade, in which cases, users should add the ones they choose to ignore to that variable. (Stricter kubeadm validation (config and runtime checks) #11710, @VannTen)

Changes by Kind

Container-Managers

• Containerd upgrade to 1.7.24 (Bump: Containerd upgrade to 1.7.24 & runc upgrade to v1.2.3 #11833, @tico88612)
  Add hashes containerd 1.7.23 (Bump containerd to 1.7.23 #11642, @robertvolkmann)
  Add hashes containerd 1.7.22 default (update containerd 1.7.22 #11554, @foobaar)
• Runc upgrade to v1.2.3 (Bump: Containerd upgrade to 1.7.24 & runc upgrade to v1.2.3 #11833, @tico88612)
  Runc add version v1.1.14 ([runc] Make runc v1.1.14 default #11574, @janosbabik)

API Change

• If you use CRI-O and want to keep runc as your container default runtime when you upgrade cluster, you must set runc_enable: true and crio_default_runtime: "runc".
  Make CRI-O's default runtime configurable
  CRI-O v1.31 default runtime change to crun
  Crun upgrade to 1.17
  Skopeo upgrade to v1.16.1 (Feat: CRI-O v1.31 change default runtime to crun #11601, @tico88612)

Feature

• Make Kubernetes v1.31.4 default
  Add hashes for Kubernetes 1.31.4, 1.30.8 and 1.29.12 (Bump: Kubernetes default version v1.31.4 #11828, @tico88612)
  Add hashes for Kubernetes 1.31.3, 1.30.7 and 1.29.11 (Bump: Kubernetes default version v1.31.3 #11737, @tico88612)
  Add hashes for Kubernetes 1.31.2, 1.30.6 and 1.29.10 (Add hashes for Kubernetes 1.31.2 (new default), 1.30.6 and 1.29.10 #11662, @robertvolkmann)
  Add hashes for Kubernetes 1.31.1 and 1.31.0 ([kubernetes] Support kubernetes 1.31.1 #11533, @philipsabri)
  Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 ([kubernetes] Add hashes for kubernetes 1.29.8, 1.29.9, 1.30.5 #11581, @DirkTheDaring)
• Add CI for openeuler 24.03
  Add CI Image for openeuler 24.03, 22.03 (Add CI for openEuler 24.03 #11689, @yankay)
• Add ResourceQuota AdmissionController plugin Configuration (Add ResourceQuota plugin configuration #11814, @chadswen)
• Add a new CRI-O crio_root variable (Add a new crio_root variable in order to store CRI-O data on somethin… #11692, @toliger)
• Add external Oracle cloud infrastructure cloud controller manager (Feat: Add external OCI cloud controller manager #11378, @tico88612)
• Add optional support for Host Firewall and PolicyAuditMode features in Cilium (Added variables cilium_enable_host_firewall and cilium_policy_audit_mode for configmap/cilium-config #11230, @ledroide)
• Add support Fedora 39/40 (Feat: support Fedora 39/40 #11573, @tico88612)
• Add support to use existing fips with terraform OpenStack (Add support to use exisitng fips with terraform openstack #11558, @anders-elastisys)
• Add the support of network isolation configuration in Multus. (Multus configuration add namespace isolation #11605, @Sispheor)
• Added support for using ntpsec (Add support for ntpsec #11665, @davidumea)
• Adds ingress_nginx_service_annotations variable to allow setting annotations for ingress-nginx controller service (Allow setting annotations on ingress-nginx service #11544, @ThisIsQasim)
• Adds nodelocaldns_additional_configs variable (Adds nodelocaldns_additional_configs to facilitate additional CoreDNS config options #11657, @0x4c6565)
• Allow disabling cilium hubble-ui using cilium_enable_hubble_ui variable (add cilium hubble-ui enable flag #10939, @pedro-peter)
• Allow to skip network configuration by setting kube_network_plugin value to none (Add option to skip network plugin installation #11844, @ant31)
• Configuration can now be supplied to ImagePolicyWebhook and PodNodeSelector admission plugins (Update the list of admission plugins which needs config #11471, @VannTen)
• Feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition (feat(calico): add support for numAllowedLocalASNumbers on bgppeers per node definition #11570, @mirwan)
• Feat: Kubeadm config API support v1beta4 (Feat: kubeadm v1beta4 support #11674, @tico88612)
• Iproute is installed before gathering facts (needed for getting ansible_default_ipv4) (Add iproute(2) package checking #11816, @0ekk)
• Partial Support of Cilium v1.16+ - kube-proxy replacement var changes
  Add optional support for configuring BGP Control Plane, IP Load Balancer Pools , Legacy BGP Peer Config v1 and BGP Config v2 features in Cilium (Partial Cilium 1.16+ Support & Add vars for configuring cilium IP load balancer pools and bgp v1 & v2 apis #11620, @logicsys)
• [cilium] Make cilium 1.15.9 default (update cilium to 1.15.9 #11593, @foobaar)
• Make cri-dockerd log level configurable (Make cri-dockerd log level configurable #11646, @mirwan)
• Remove support Fedora 37/38 (Cleanup: Fedora 37/38 CI tests & docs #11600, @tico88612)
• Reset operation: remove /var/log/containers and disable service auto-boot, make sure that multi-user.target.wants is deleted. (Make sure the Reset cleanup is more complete #11501, @leeonfu)
• Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 (Support Configuring EncryptionAlgorithm in Kubeadm v1beta4 #11757, @ErikJiang)
• Update crictl to version v1.31.1 for Kubernetes 1.31
  Update crictl to version v1.30.1 for Kubernetes 1.30 (Add hashes for crictl 1.30.1 and 1.31.1 #11661, @robertvolkmann)
• Update multus to v4.1.0 (Update multus to v4.1.0 and clarify cilium compatibility #11434, @ThisIsQasim)
• Upgrade CoreDNS version to v1.11.3 (Feat: CoreDNS upgrade to v1.11.3 #11653, @tico88612)
• Upgrade OpenStack Cloud Controller Manager to v1.31.1 (Bump: OpenStack Cloud Controller Manager to v1.31.1 #11738, @tico88612)
• Upgrade pause container to 3.10 (Bump: pause container upgrade to 3.10 #11695, @tico88612)
• [calico] Update default calico to v3.29.1 ([calico] Add version 3.29.1 and make it default #11798, @mzaian)
• [cert-manager] upgrade to v1.15.3 ([cert-manager] upgrade to v1.15.3 #11668, @tico88612)
• [cri-o] Switch binaries to libexecdir
  Update youki version to 0.4.1 to fix ci. ([cri-o] Switch binaries to libexecdir #11584, @yankay)
• [etcd] Default version to 3.5.16 for 1.28, 1.29, 1.30, 1.31 ([etcd] make etcd 3.5.16 default #11572, @janosbabik)
• [helm] Upgrade to v3.16.4, add 3.16.x checksum (Bump: Helm upgrade to v3.16.4 #11832, @tico88612)
• [ingress-nginx] upgrade controller to version 1.12.0 ([ingress-nginx] upgrade to 1.12.0 #11846, @mzaian)
• [need notice] update containerd max_container_log_line_size default value to 16384 (Update containerd containerd_max_container_log_line_size default value to 16384 #11585, @KubeKyrie)
• [nerdctl] Default version to 1.7.7 ([nerdctl] Make nerdctl 1.7.7 default #11575, @janosbabik)

Documentation

• No longer support in-tree cloud provider, please delete or write external to the cloud_provider variable. (Cleanup: remove in-tree cloud provider support #11633, @tico88612)
• Remove inventory_builder scripts and contrib/dind (Remove inventory_builder and re-organize docs #11748, @VannTen)
• Update dns-stack.md reference in docs/ansible/vars.md (Update dns-stack.md reference #11745, @emmanuel-ferdman)

Failing Test

• Fix CI: elastx cleanup security groups (Fix CI: elastx cleanup security groups  #11411, @yankay)
• Update rockylinux9 base image to 9.5 (Update rockylinux9 base image to 9.5 #11744, @yankay)

Bug or Regression

• Action required
  Running kubespray with --limit without cached facts is no longer supported. Improves the scaling for large clusters. (Do not serialize fact gathering for no_proxy #11598, @VannTen)
• Always copy cert generation script to first etcd to pick up fixes on existing clusters (Always copy cert generation scripts to first etcd #11612, @VannTen)
• Fix Cilium agent permission can't read loadbalancerippools and secrets (Update cluster-role for cilium to prevent errors in agent startup #11466, @foobaar)
• Fix calico dual stack installation when using ip and ip6. (calico: stop recording calico_kubelet_name #11770, @VannTen)
• Fix collection usage for calico and other configuration depending on .sh and .conf files in Kubespray (Include scripts and conf files in the ansible collection #11707, @VannTen)
• Fix format of kubeadm-config v1beta4 (Remove deprecated key from kubeadmconfig/v1beta4 #11709, @VannTen)
• Fix kube-vip container securityContext (fix kube-vip container securityContext #11647, @KubeKyrie)
• Fix openEuler system packages installation (Simplify package selection + fix openEuler package #11688, @VannTen)
• Fix pretty-printing (in kubectl) of nodelocaldns and coredns configmap when using dns_upstream_forward_extra_opts with an empty value option. (Fix pretty-printing of coredns and nodelocaldns configmaps #11694, @VannTen)
• Fix spurious failure with 'localhost' when using scale.yml --limit <some nodes> (Only consider host in 'k8s_cluster' when checking if ip is a cached fact #11817, @VannTen)
• Fix task naming in bootstrap-os (fix task naming in bootstrap-os #11714, @ErikJiang)
• Fix terraform.py on python >=3.12 (Support python 3.12 for terraform.py in regex expression #11773, @enrico9034)
• Fix the check for cached data when using --limit (preinstall: fix checking that excluded host are cached #11693, @VannTen)
• Fix the usage of --limit when using legacy groups (Fix dynamic groups definition when using --limit #11577, @VannTen)
• Fix usage of admission plugins configuration. (apiserver: fix incorrect path to admission plugins config files #11779, @VannTen)
• Fix using the default network manager in reset.yml (Fix using the default network manager in reset.yml #11678, @KubeKyrie)
• Fix: cannot stop & remove all cri containers via remove_node.yml (Fix: cannot stop & remove all cri containers via remove_node.yml #11631, @tico88612)
• Fixed: VSphere CSI and CPI drivers and are now retrieved from registry.k8s.io instead of gcr.io, as they have been deleted from the latter. Only a few recent versions are available in the new repository; if you have pinned vsphere_csi_controller, vsphere_csi_driver_image_tag or vsphere_syncer_image_tag to a version older than v3.1.2, please check if that version is available from the new repository. The same goes for external_vsphere_cloud_controller_image_tag which can no longer be latest, and should align with the running version of Kubernetes. It now defaults to v1.31.0. (fix: vsphere image repositories, tags and docs #11564, @luringens)
• HA etcd cluster keeps quorum during upgrades. (etcd: throttle restart for availability #11677, @VannTen)
• Kubeadm images (kube-controller-manager,kube-scheduler,kube-apiserver,kube-proxy) are properly downloaded, including when using the download cache. (Fix the format of the list of images extracted from kubeadm #11741, @VannTen)
• Make sure kubespray-defaults can be executed successfully by executing bootstrap-os first (fix: Swap roles kubespray-defaults & bootstrap-os #11441, @huangkevin404)
• Make upcloud csi_driver use the correct pull secret (Use correct secret name for csi-upcloud pull secret #11597, @VannTen)
• Modifies Helm parameters wait and atomic to be set to false when using kube_network_plugin=cni to prevent deployment issues with kubelet-csr-approver. (Fix kubelet-csr-approver deployment failes if kube_network_plugin=cni(generic) #11704, @M-JavadHeydarpour)
• Remove invalid extraArgs entry and update template file reference (remove nameless extraArgs entry #11703, @agravgaard)
• Update calico-nopde template and remove flexvol-driver initContainer (Fix calico-node template and remove flexvol-driver initContainer #11634, @KubeKyrie)
• Use correct version for community.general collection (Bump version of community.general #11724, @VannTen)

Other (Cleanup or Flake)

• Cleanup older terminology, replace "master" with "control plane" (Cleanup master terminology (start fixing issue #11353) #11394, @bogd)
• Drop support for Kubernetes 1.28.x minimum version now is 1.29.x
  Drop support for CRI-O 1.28.x minimum version now is 1.29.x (Drop support for Kubernetes 1.28.x #11609, @yankay)
• Fix roles/download/tasks/download_file.yml task name typo (Fix task name spelling #11684, @dmncmn)
• Optimize CA cert hash calculation with community.crypto (Optimize CA cert hash calculation with community.crypto #11758, @ErikJiang)
• Remove pip install . support and rpm spec file (Remove pip and rpm packaging support #11760, @VannTen)
• Replace deprecated unarchive.copy with unarchive.remote_src (replace deprecated unarchive.copy with unarchive.remote_src #11207, @Payback159)
• Update KUBESPRAY_VERSION for v2.26.0 (Update KUBESPRAY_VERSION in CI and Readme for v2.26.0 #11511, @yankay)
• containerd_use_config_path is removed as kubespray now always use containerd config_path configuration. (containerd: always use config_path #11755, @VannTen)

Tasks

Preview Give feedback

1. Add iproute(2) package checking #11816
   approved cncf-cla: yes kind/feature lgtm ok-to-test release-note size/M +7
   
2. Only consider host in 'k8s_cluster' when checking if ip is a cached fact #11817
   approved cncf-cla: yes kind/bug lgtm ok-to-test release-note size/XS +7
   
3. Bump: Kubernetes default version v1.31.4 #11828
   approved cncf-cla: yes kind/feature lgtm ok-to-test release-note size/M +7
   
4. [calico] Add version 3.29.1 and make it default #11798
   approved cncf-cla: yes kind/feature lgtm ok-to-test release-note size/L +7
   
5. Bump: Helm upgrade to v3.16.4 #11832
   approved cncf-cla: yes kind/feature lgtm ok-to-test release-note size/S +7
   
6. Bump: Containerd upgrade to 1.7.24 & runc upgrade to v1.2.3 #11833
   approved cncf-cla: yes kind/container-managers kind/feature lgtm ok-to-test release-note size/S +8
   

[VannTen]

Hum I need to fix some of my release notes it seems

[VannTen]

I've updated the release note of my PR, the rendering should be better now.

[yankay]

The Kube 1.32 has been released. https://github.com/kubernetes/kubernetes/releases/tag/v1.32.0
So we need to release kubespray in a few weeks :-)

HI @tico88612
Please update the release notes and create a list of pending issues?
If the pending issue has been fixed, the kubespray will be OK to release :-)

[tico88612]

@yankay I will review the PR & reorganize the release note in this week.

[VannTen]

#11808 Should go in before the release (to unbreak hardening)

[VannTen]

(EDIT: -> put the fixes in the tasklist (that feature is pretty good 🤔 )

[yankay]

It needs some bumps

• helm from v3.15.4 to v3.16.4
• containerd from v1.7.23 to 1.7.24
• runc from v1.1.14 to v1.2.3 ?
• kubernetes from v1.31.3 to v1.31.4

[tico88612]

Hi maintainers @yankay @VannTen @mzaian

Are there any critical PRs that need to be on v2.27?
The packages (Kubernetes, runc, contained, helm) have now been updated to the latest version.

[mzaian]

Hi maintainers @yankay @VannTen @mzaian

Are there any critical PRs that need to be on v2.27? The packages (Kubernetes, runc, contained, helm) have now been updated to the latest version.

I will have a look at other things and let you know!

[VannTen]

I don't see anything critical as of now, but I'm only back at work next week ; I'll take a more thorough look then (no need to wait for me if everyone else is ok with the current state).

[yankay]

There seems no block issue found by me. So I'm ready to release it next week( before 2024-01-11)

[tico88612]

@VannTen PR #11813 Do we have to do it in this release v2.27? Or should we wait until the next release?

[VannTen]

It can wait

[tico88612]

Hello approvers @mzaian @yankay @VannTen @ant31,

Kubespray v2.27's release note is ready. Could you take a look? (｡･ω･｡)ﾉ
If there are no problems, we're ready to release v2.27.0.

[yankay]

Hello approvers @mzaian @yankay @VannTen @ant31,

Kubespray v2.27's release note is ready. Could you take a look? (｡･ω･｡)ﾉ If there are no problems, we're ready to release v2.27.0.

Thanks for all.
I'll release it in 3 days.