Add kubeadm option to etcd_deployment_type to replace the etcd_kubead…

…m_enabled variable (kubernetes-sigs#8317) * Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable Signed-off-by: necatican <[email protected]> * Add etcd kubeadm deployment documentation Signed-off-by: necatican <[email protected]> * Refactor warning for the deprecated 'etcd_kubeadm_enabled' variable Signed-off-by: necatican <[email protected]>
southbridgeio · Oct 21, 2023 · a814519 · a814519
1 parent 7dbc0f8
commit a814519
Show file tree

Hide file tree

Showing 30 changed files with 135 additions and 80 deletions.
diff --git a/cluster.yml b/cluster.yml
@@ -46,7 +46,7 @@
       vars:
         etcd_cluster_setup: true
         etcd_events_cluster_setup: "{{ etcd_events_cluster_enabled }}"
-      when: not etcd_kubeadm_enabled| default(false)
+      when: etcd_deployment_type != "kubeadm"
 
 - hosts: k8s_cluster
   gather_facts: False
@@ -59,7 +59,7 @@
       vars:
         etcd_cluster_setup: false
         etcd_events_cluster_setup: false
-      when: not etcd_kubeadm_enabled| default(false)
+      when: etcd_deployment_type != "kubeadm"
 
 - hosts: k8s_cluster
   gather_facts: False

diff --git a/docs/cri-o.md b/docs/cri-o.md
@@ -14,7 +14,7 @@ _To use the CRI-O container runtime set the following variables:_
 kubeadm_enabled: true
 download_container: false
 skip_downloads: false
-etcd_kubeadm_enabled: true
+etcd_deployment_type: host # optionally kubeadm
 ```
 
 ## k8s_cluster/k8s_cluster.yml
@@ -23,12 +23,6 @@ etcd_kubeadm_enabled: true
 container_manager: crio
 ```
 
-## etcd.yml
-
-```yaml
-etcd_deployment_type: host # optionally and mutually exclusive with etcd_kubeadm_enabled
-```
-
 ## all/crio.yml
 
 Enable docker hub registry mirrors

diff --git a/docs/etcd.md b/docs/etcd.md
@@ -1,5 +1,21 @@
 # etcd
 
+## Deployment Types
+
+It is possible to deploy etcd with three methods. To change the default deployment method (host), use the `etcd_deployment_type` variable. Possible values are `host`, `kubeadm`, and `docker`.
+
+### Host
+
+Host deployment is the default method. Using this method will result in etcd installed as a systemd service.
+
+### Docker
+
+Installs docker in etcd group members and runs etcd on docker containers. Only usable when `container_manager` is set to `docker`.
+
+### Kubeadm
+
+This deployment method is experimental and is only available for new deployments. This deploys etcd as a static pod in master hosts.
+
 ## Metrics
 
 To expose metrics on a separate HTTP port, define it in the inventory with:

diff --git a/docs/upgrades.md b/docs/upgrades.md
@@ -308,6 +308,18 @@ caprica   Ready    master,node   7h40m   v1.14.1
 
 ```
 
+## Upgrading to v2.19
+
+`etcd_kubeadm_enabled` is being deprecated at v2.19. The same functionality is achievable by setting `etcd_deployment_type` to `kubeadm`.
+Deploying etcd using kubeadm is experimental and is only available for either new or deployments where `etcd_kubeadm_enabled` was set to `true` while deploying the cluster.
+
+From 2.19 and onward `etcd_deployment_type` variable will be placed in `group_vars/all/etcd.yml` instead of `group_vars/etcd.yml`, due to scope issues.
+The placement of the variable is only important for `etcd_deployment_type: kubeadm` right now. However, since this might change in future updates, it is recommended to move the variable.
+
+Upgrading is straightforward; no changes are required if `etcd_kubeadm_enabled` was not set to `true` when deploying.
+
+If you have a cluster where `etcd` was deployed using `kubeadm`, you will need to remove `etcd_kubeadm_enabled` the variable. Then move `etcd_deployment_type` variable from `group_vars/etcd.yml` to `group_vars/all/etcd.yml` due to scope issues and set `etcd_deployment_type` to `kubeadm`.
+
 ## Upgrade order
 
 As mentioned above, components are upgraded in the order in which they were

diff --git a/inventory/prod/group_vars/all/all.yml b/inventory/prod/group_vars/all/all.yml
@@ -2,12 +2,6 @@
 # min supported kube_version
 # kube_version_min_required: v1.18.0
 
-## Directory where etcd data stored
-etcd_data_dir: /var/lib/etcd
-
-## Experimental kubeadm etcd deployment mode. Available only for new deployment
-etcd_kubeadm_enabled: false
-
 ## Directory where the binaries will be installed
 bin_dir: /usr/local/bin
 
@@ -18,7 +12,6 @@ bin_dir: /usr/local/bin
 ## but don't know about that address themselves.
 # access_ip: 1.1.1.1
 
-
 ## External LB example config
 ## apiserver_loadbalancer_domain_name: "elb.some.domain"
 # loadbalancer_apiserver:

diff --git a/inventory/prod/group_vars/all/etcd.yml b/inventory/prod/group_vars/all/etcd.yml
@@ -0,0 +1,17 @@
+---
+## Directory where etcd data stored
+etcd_data_dir: /var/lib/etcd
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
+## Kubeadm etcd deployment is experimental and only available for new deployments
+## If this is not set, container manager will be inherited from the Kubespray defaults
+## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
+## Also this makes possible to use different container manager for etcd nodes.
+# container_manager: containerd
+
+## Settings for etcd deployment type
+# Set this to docker if you are using container_manager: docker
+# may be host, docker, kubeadm
+etcd_deployment_type: host
diff --git a/inventory/prod/group_vars/etcd.yml b/inventory/prod/group_vars/etcd.yml
@@ -18,6 +18,5 @@ etcd_memory_limit: 0
 # etcd_peer_client_auth: true
 
 etcd_snapshot_count: "5000"
-etcd_ionice: "-c2 -n0"
 
-etcd_deployment_type: docker
+etcd_deployment_type: host
diff --git a/inventory/s000/group_vars/all/all.yml b/inventory/s000/group_vars/all/all.yml
@@ -2,12 +2,6 @@
 # min supported kube_version
 # kube_version_min_required: v1.18.0
 
-## Directory where etcd data stored
-etcd_data_dir: /var/lib/etcd
-
-## Experimental kubeadm etcd deployment mode. Available only for new deployment
-etcd_kubeadm_enabled: false
-
 ## Directory where the binaries will be installed
 bin_dir: /usr/local/bin
 
@@ -18,7 +12,6 @@ bin_dir: /usr/local/bin
 ## but don't know about that address themselves.
 # access_ip: 1.1.1.1
 
-
 ## External LB example config
 ## apiserver_loadbalancer_domain_name: "elb.some.domain"
 # loadbalancer_apiserver:

diff --git a/inventory/s000/group_vars/all/etcd.yml b/inventory/s000/group_vars/all/etcd.yml
@@ -0,0 +1,17 @@
+---
+## Directory where etcd data stored
+etcd_data_dir: /var/lib/etcd
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
+## Kubeadm etcd deployment is experimental and only available for new deployments
+## If this is not set, container manager will be inherited from the Kubespray defaults
+## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
+## Also this makes possible to use different container manager for etcd nodes.
+# container_manager: containerd
+
+## Settings for etcd deployment type
+# Set this to docker if you are using container_manager: docker
+# may be host, docker, kubeadm
+etcd_deployment_type: host
diff --git a/inventory/s000/group_vars/etcd.yml b/inventory/s000/group_vars/etcd.yml
@@ -19,7 +19,4 @@ etcd_memory_limit: 0
 
 etcd_snapshot_count: "5000"
 
-# ionice removed from container etcd:3.4.x
-#etcd_ionice: "-c2 -n0"
-
-etcd_deployment_type: docker
+etcd_deployment_type: host
diff --git a/inventory/sample/group_vars/all/all.yml b/inventory/sample/group_vars/all/all.yml
@@ -1,10 +1,4 @@
 ---
-## Directory where etcd data stored
-etcd_data_dir: /var/lib/etcd
-
-## Experimental kubeadm etcd deployment mode. Available only for new deployment
-etcd_kubeadm_enabled: false
-
 ## Directory where the binaries will be installed
 bin_dir: /usr/local/bin
 

diff --git a/inventory/sample/group_vars/all/etcd.yml b/inventory/sample/group_vars/all/etcd.yml
@@ -0,0 +1,16 @@
+---
+## Directory where etcd data stored
+etcd_data_dir: /var/lib/etcd
+
+## Container runtime
+## docker for docker, crio for cri-o and containerd for containerd.
+## Additionally you can set this to kubeadm if you want to install etcd using kubeadm
+## Kubeadm etcd deployment is experimental and only available for new deployments
+## If this is not set, container manager will be inherited from the Kubespray defaults
+## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
+## Also this makes possible to use different container manager for etcd nodes.
+# container_manager: containerd
+
+## Settings for etcd deployment type
+# Set this to docker if you are using container_manager: docker
+etcd_deployment_type: host
diff --git a/inventory/sample/group_vars/etcd.yml b/inventory/sample/group_vars/etcd.yml
@@ -17,14 +17,3 @@
 ### ETCD: disable peer client cert authentication.
 # This affects ETCD_PEER_CLIENT_CERT_AUTH variable
 # etcd_peer_client_auth: true
-
-## Container runtime
-## docker for docker, crio for cri-o and containerd for containerd.
-## If this is not set, container manager will be inherited from the Kubespray defaults
-## and not from k8s_cluster/k8s-cluster.yml, which might not be what you want.
-## Also this makes possible to use different container manager for etcd nodes.
-# container_manager: containerd
-
-## Settings for etcd deployment type
-# Set this to docker if you are using container_manager: docker
-etcd_deployment_type: host
diff --git a/roles/download/templates/kubeadm-images.yaml.j2 b/roles/download/templates/kubeadm-images.yaml.j2
@@ -8,7 +8,7 @@ kind: ClusterConfiguration
 imageRepository: {{ kube_image_repo }}
 kubernetesVersion: {{ kube_version }}
 etcd:
-{% if etcd_kubeadm_enabled %}
+{% if etcd_deployment_type == "kubeadm" %}
   local:
     imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}"
     imageTag: "{{ etcd_image_tag }}"

diff --git a/roles/etcdctl/tasks/main.yml b/roles/etcdctl/tasks/main.yml
@@ -4,7 +4,7 @@
 
 - name: Check unintentional include of this role
   assert:
-    that: etcd_kubeadm_enabled
+    that: etcd_deployment_type == "kubeadm"
 
 - name: Check if etcdctl exist
   stat:

diff --git a/roles/kubernetes/control-plane/defaults/main/main.yml b/roles/kubernetes/control-plane/defaults/main/main.yml
@@ -2,9 +2,6 @@
 # disable upgrade cluster
 upgrade_cluster_setup: false
 
-# Experimental kubeadm etcd deployment mode. Available only for new deployment
-etcd_kubeadm_enabled: false
-
 # An experimental dev/test only dynamic volumes provisioner,
 # for PetSets. Works for kube>=v1.3 only.
 kube_hostpath_dynamic_provisioner: "false"

diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-etcd.yml b/roles/kubernetes/control-plane/tasks/kubeadm-etcd.yml
@@ -15,4 +15,4 @@
 - name: Ensure etcdctl script is installed
   import_role:
     name: etcdctl
-  when: etcd_kubeadm_enabled
+  when: etcd_deployment_type == "kubeadm"
diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-fix-apiserver.yml b/roles/kubernetes/control-plane/tasks/kubeadm-fix-apiserver.yml
@@ -21,4 +21,4 @@
     dest: "{{ kube_config_dir }}/manifests/kube-apiserver.yaml"
     regexp: '^    - --etcd-servers='
     line: '    - --etcd-servers={{ etcd_access_addresses }}'
-  when: not etcd_kubeadm_enabled | default(false)
+  when: etcd_deployment_type != "kubeadm"
diff --git a/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml b/roles/kubernetes/control-plane/tasks/kubeadm-upgrade.yml
@@ -18,7 +18,7 @@
     --config={{ kube_config_dir }}/kubeadm-config.yaml
     --ignore-preflight-errors=all
     --allow-experimental-upgrades
-    --etcd-upgrade={{ etcd_kubeadm_enabled | bool | lower }}
+    --etcd-upgrade={{ etcd_deployment_type == "kubeadm" | bool | lower }}
     --force
   register: kubeadm_upgrade
   # Retry is because upload config sometimes fails
@@ -39,7 +39,7 @@
     --config={{ kube_config_dir }}/kubeadm-config.yaml
     --ignore-preflight-errors=all
     --allow-experimental-upgrades
-    --etcd-upgrade={{ etcd_kubeadm_enabled | bool | lower }}
+    --etcd-upgrade={{ etcd_deployment_type == "kubeadm" | bool | lower }}
     --force
   register: kubeadm_upgrade
   when: inventory_hostname != first_kube_control_plane

diff --git a/roles/kubernetes/control-plane/tasks/main.yml b/roles/kubernetes/control-plane/tasks/main.yml
@@ -76,7 +76,7 @@
   include_tasks: kubeadm-etcd.yml
   when:
     - kubeadm_enabled|bool|default(false)
-    - etcd_kubeadm_enabled
+    - etcd_deployment_type == "kubeadm"
 
 - name: Include kubeadm secondary server apiserver fixes
   include_tasks: kubeadm-fix-apiserver.yml

diff --git a/roles/kubernetes/control-plane/templates/kubeadm-config.v1beta2.yaml.j2 b/roles/kubernetes/control-plane/templates/kubeadm-config.v1beta2.yaml.j2
@@ -33,7 +33,7 @@ apiVersion: kubeadm.k8s.io/v1beta2
 kind: ClusterConfiguration
 clusterName: {{ cluster_name }}
 etcd:
-{% if not etcd_kubeadm_enabled %}
+{% if etcd_deployment_type != "kubeadm" %}
   external:
       endpoints:
 {% for endpoint in etcd_access_addresses.split(',') %}
@@ -42,7 +42,7 @@ etcd:
       caFile: {{ etcd_cert_dir }}/{{ kube_etcd_cacert_file }}
       certFile: {{ etcd_cert_dir }}/{{ kube_etcd_cert_file }}
       keyFile: {{ etcd_cert_dir }}/{{ kube_etcd_key_file }}
-{% elif etcd_kubeadm_enabled %}
+{% elif etcd_deployment_type == "kubeadm" %}
   local:
     imageRepository: "{{ etcd_image_repo | regex_replace("/etcd$","") }}"
     imageTag: "{{ etcd_image_tag }}"

diff --git a/roles/kubernetes/kubeadm/defaults/main.yml b/roles/kubernetes/kubeadm/defaults/main.yml
@@ -10,6 +10,3 @@ kube_override_hostname: >-
   {%- else -%}
   {{ inventory_hostname }}
   {%- endif -%}
-
-# Experimental kubeadm etcd deployment mode. Available only for new deployment
-etcd_kubeadm_enabled: false
diff --git a/roles/kubernetes/kubeadm/tasks/main.yml b/roles/kubernetes/kubeadm/tasks/main.yml
@@ -166,7 +166,7 @@
 - name: Extract etcd certs from control plane if using etcd kubeadm mode
   include_tasks: kubeadm_etcd_node.yml
   when:
-    - etcd_kubeadm_enabled
+    - etcd_deployment_type == "kubeadm"
     - inventory_hostname not in groups['kube_control_plane']
     - kube_network_plugin in ["calico", "flannel", "canal", "cilium"] or cilium_deploy_additionally | default(false) | bool
     - kube_network_plugin != "calico" or calico_datastore == "etcd"
diff --git a/roles/kubernetes/preinstall/tasks/0020-verify-settings.yml b/roles/kubernetes/preinstall/tasks/0020-verify-settings.yml
@@ -15,7 +15,7 @@
   run_once: true
   when:
     - not ignore_assert_errors
-    - not etcd_kubeadm_enabled
+    - etcd_deployment_type != "kubeadm"
 
 - name: Stop if non systemd OS type
   assert:
@@ -287,23 +287,41 @@
   when: resolvconf_mode is defined
   run_once: true
 
-- name: Stop if etcd deployment type is not host or docker
+- name: Stop if etcd deployment type is not host, docker or kubeadm
   assert:
-    that: etcd_deployment_type in ['host', 'docker']
-    msg: "The etcd deployment type, 'etcd_deployment_type', must be host or docker"
+    that: etcd_deployment_type in ['host', 'docker', 'kubeadm']
+    msg: "The etcd deployment type, 'etcd_deployment_type', must be host, docker or kubeadm"
   when:
     - inventory_hostname in groups.get('etcd',[])
-    - not etcd_kubeadm_enabled
 
-- name: Stop if etcd deployment type is not host when container_manager != docker
+- name: Stop if etcd deployment type is not host or kubeadm when container_manager != docker
   assert:
-    that: etcd_deployment_type == 'host'
-    msg: "The etcd deployment type, 'etcd_deployment_type', must be host when container_manager is not docker"
+    that: etcd_deployment_type in ['host', 'kubeadm']
+    msg: "The etcd deployment type, 'etcd_deployment_type', must be host or kubeadm when container_manager is not docker"
   when:
     - inventory_hostname in groups.get('etcd',[])
-    - not etcd_kubeadm_enabled
     - container_manager != 'docker'
 
+# TODO: Clean this task up when we drop backward compatibility support for `etcd_kubeadm_enabled`
+- name: Stop if etcd deployment type is not host or kubeadm when container_manager != docker and etcd_kubeadm_enabled is not defined
+  block:
+    - name: Warn the user if they are still using `etcd_kubeadm_enabled`
+      debug:
+        msg: >
+          "WARNING! => `etcd_kubeadm_enabled` is deprecated and will be removed in a future release.
+          You can set `etcd_deployment_type` to `kubeadm` instead of setting `etcd_kubeadm_enabled` to `true`."
+      changed_when: true
+
+    - name: Stop if `etcd_kubeadm_enabled` is defined and `etcd_deployment_type` is not `kubadm` or `host`
+      assert:
+        that: etcd_deployment_type == 'kubeadm'
+        msg: >
+          It is not possible to use `etcd_kubeadm_enabled` when `etcd_deployment_type` is set to {{ etcd_deployment_type }}.
+          Unset the `etcd_kubeadm_enabled` variable and set `etcd_deployment_type` to desired deployment type (`host`, `kubeadm`, `docker`) instead."
+      when: etcd_kubeadm_enabled
+  run_once: yes
+  when: etcd_kubeadm_enabled is defined
+
 - name: Stop if download_localhost is enabled but download_run_once is not
   assert:
     that: download_run_once

diff --git a/roles/kubernetes/preinstall/tasks/0040-set_facts.yml b/roles/kubernetes/preinstall/tasks/0040-set_facts.yml
@@ -215,7 +215,7 @@
     kube_etcd_cert_file: "apiserver-etcd-client.crt"
     kube_etcd_key_file: "apiserver-etcd-client.key"
   when:
-    - etcd_kubeadm_enabled
+    - etcd_deployment_type == "kubeadm"
 
 - name: check /usr readonly
   stat: