Overhaul Cilium manifests to match the newer versions (kubernetes-sig…

…s#8717)

* [cilium] Separate templates for cilium, cilium-operator, and hubble installations

Signed-off-by: necatican <[email protected]>

* [cilium] Update cilium-operator templates

Signed-off-by: necatican <[email protected]>

* [cilium] Allow using custom args and mounting extra volumes for the Cilium Operator

Signed-off-by: necatican <[email protected]>

* [cilium] Update the cilium configmap to filter out the deprecated variables, and add the new variables

Signed-off-by: necatican <[email protected]>

* [cilium] Add an option to use Wireguard encryption on Cilium 1.10 and up

Signed-off-by: necatican <[email protected]>

* [cilium] Update cilium-agent templates

Signed-off-by: necatican <[email protected]>

* [cilium] Bump Cilium version to 1.11.3

Signed-off-by: necatican <[email protected]>

docs/cilium.md

@@ -12,10 +12,51 @@ the external loadbalancer (even from a node in the control plane)
 and if there is no external load balancer It will ignore any local load
 balancer deployed by Kubespray and **only contacts the first master**.
 
+## Cilium Operator
+
+Unlike some operators, Cilium Operator does not exist for installation purposes.
+> The Cilium Operator is responsible for managing duties in the cluster which should logically be handled once for the entire cluster, rather than once for each node in the cluster.
+
+### Adding custom flags to the Cilium Operator
+
+You can set additional cilium-operator container arguments using `cilium_operator_custom_args`.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+Accepts an array or a string.
+
+```yml
+cilium_operator_custom_args: ["--foo=bar", "--baz=qux"]
+```
+
+or
+
+```yml
+cilium_operator_custom_args: "--foo=bar"
+```
+
+You do not need to add a custom flag to enable debugging. Instead, feel free to use the `CILIUM_DEBUG` variable.
+
+### Adding extra volumes and mounting them
+
+You can use `cilium_operator_extra_volumes` to add extra volumes to the Cilium Operator, and use `cilium_operator_extra_volume_mounts` to mount those volumes.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+```yml
+cilium_operator_extra_volumes:
+  - configMap:
+      name: foo
+    name: foo-mount-path
+
+cilium_operator_extra_volume_mounts:
+  - mountPath: /tmp/foo/bar
+    name: foo-mount-path
+    readOnly: true
+```
+
 ## Choose Cilium version
 
 ```yml
-cilium_version: v1.11.0
+cilium_version: v1.11.3
 ```
 
 ## Add variable to config
@@ -39,6 +80,47 @@ Cilium currently supports two different identity allocation modes:
   - These can be queried with `kubectl get ciliumid`
 - "kvstore" stores identities in an etcd kvstore.
 
+## Enable Transparent Encryption
+
+Cilium supports the transparent encryption of Cilium-managed host traffic and
+traffic between Cilium-managed endpoints either using IPsec or Wireguard.
+
+Wireguard option is only available in Cilium 1.10.0 and newer.
+
+### IPsec Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/)
+
+To enable IPsec encryption, you just need to set three variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "ipsec"
+```
+
+The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable.
+Kubespray does not automate this process.
+Cilium documentation currently recommends creating a key using the following command:
+
+```shell
+echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"
+```
+
+Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable.
+
+### Wireguard Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/)
+
+To enable Wireguard encryption, you just need to set two variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "wireguard"
+```
+
+Kubespray currently supports Linux distributions with Wireguard Kernel mode on Linux 5.6 and newer.
+
 ## Install Cilium Hubble
 
 k8s-net-cilium.yml:
@@ -68,6 +150,6 @@ cilium_hubble_metrics:
   - flow
   - icmp
   - http
-```  
+```
 
 [More](https://docs.cilium.io/en/v1.9/operations/metrics/#hubble-exported-metrics)

inventory/sample/group_vars/k8s_cluster/k8s-net-cilium.yml

@@ -1,4 +1,47 @@
 # see roles/network_plugin/cilium/defaults/main.yml
 
-# cilium_version: "v1.11.0"
+# cilium_version: "v1.11.3"
 # cilium_identity_allocation_mode: kvstore # kvstore or crd
+
+# For adding and mounting extra volumes to the cilium operator
+# cilium_operator_extra_volumes: []
+# cilium_operator_extra_volume_mounts: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+# cilium_cluster_name: default
+
+# Unique ID of the cluster. Must be unique across all conneted clusters and
+# in the range of 1 and 255. Only relevant when building a mesh of clusters.
+# This value is not defined by default
+# cluster-id:
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
+# cilium_native_routing_cidr: ""
+
+# Allows to explicitly specify the IPv6 CIDR for native routing.
+# cilium_native_routing_cidr_ipv6: ""
+
+# Encryption
+# Enable transparent network encryption.
+# cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+# cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+# cilium_ipsec_node_encryption: "false"
+
+# Enables the fallback to the user-space implementation.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+# cilium_wireguard_userspace_fallback: "false"

roles/download/defaults/main.yml

@@ -110,7 +110,7 @@ flannel_cni_version: "v1.0.1"
 cni_version: "v1.0.1"
 weave_version: 2.8.1
 pod_infra_version: "3.3"
-cilium_version: "v1.11.1"
+cilium_version: "v1.11.3"
 kube_ovn_version: "v1.8.1"
 kube_router_version: "v1.4.0"
 multus_version: "v3.8"

roles/network_plugin/cilium/defaults/main.yml

@@ -48,7 +48,11 @@ cilium_kube_proxy_replacement: probe
 # to prevent service disruptions. See also:
 # http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
 cilium_preallocate_bpf_maps: false
+
+# `cilium_tofqdns_enable_poller` is deprecated in 1.8, removed in 1.9
 cilium_tofqdns_enable_poller: false
+
+# `cilium_enable_legacy_services` is deprecated in 1.6, removed in 1.9
 cilium_enable_legacy_services: false
 
 # Deploy cilium even if kube_network_plugin is not cilium.
@@ -62,10 +66,38 @@ cilium_deploy_additionally: false
 # make this work. Please refer to the cilium documentation for more
 # information about this kind of setups.
 cilium_auto_direct_node_routes: false
+
+# Allows to explicitly specify the IPv4 CIDR for native routing.
+# When specified, Cilium assumes networking for this CIDR is preconfigured and
+# hands traffic destined for that range to the Linux network stack without
+# applying any SNAT.
+# Generally speaking, specifying a native routing CIDR implies that Cilium can
+# depend on the underlying networking stack to route packets to their
+# destination. To offer a concrete example, if Cilium is configured to use
+# direct routing and the Kubernetes CIDR is included in the native routing CIDR,
+# the user must configure the routes to reach pods, either manually or by
+# setting the auto-direct-node-routes flag.
 cilium_native_routing_cidr: ""
 
-# IPsec based transparent encryption between nodes
-cilium_ipsec_enabled: false
+# Allows to explicitly specify the IPv6 CIDR for native routing.
+cilium_native_routing_cidr_ipv6: ""
+
+# Enable transparent network encryption.
+cilium_encryption_enabled: false
+
+# Encryption method. Can be either ipsec or wireguard.
+# Only effective when `cilium_encryption_enabled` is set to true.
+cilium_encryption_type: "ipsec"
+
+# Enable encryption for pure node to node traffic.
+# This option is only effective when `cilium_encryption_type` is set to `ipsec`.
+cilium_ipsec_node_encryption: "false"
+
+# If your kernel or distribution does not support WireGuard, Cilium agent can be configured to fall back on the user-space implementation.
+# When this flag is enabled and Cilium detects that the kernel has no native support for WireGuard,
+# it will fallback on the wireguard-go user-space implementation of WireGuard.
+# This option is only effective when `cilium_encryption_type` is set to `wireguard`.
+cilium_wireguard_userspace_fallback: "false"
 
 # Hubble
 ### Enable Hubble without install
@@ -89,6 +121,15 @@ cilium_hubble_tls_generate: false
 # https://docs.cilium.io/en/v1.9/concepts/networking/ipam/
 cilium_ipam_mode: kubernetes
 
+# Extra arguments for the Cilium agent
+cilium_agent_custom_args: []
+
+# For adding and mounting extra volumes to the cilium agent
+cilium_agent_extra_volumes: []
+cilium_agent_extra_volume_mounts: []
+
+cilium_agent_extra_env_vars: []
+
 # The address at which the cillium operator bind health check api
 cilium_operator_api_serve_addr: "127.0.0.1:9234"
 
@@ -97,3 +138,23 @@ cilium_operator_api_serve_addr: "127.0.0.1:9234"
 ##    var1: "value1"
 ##    var2: "value2"
 cilium_config_extra_vars: {}
+
+# For adding and mounting extra volumes to the cilium operator
+cilium_operator_extra_volumes: []
+cilium_operator_extra_volume_mounts: []
+
+# Extra arguments for the Cilium Operator
+cilium_operator_custom_args: []
+
+# Name of the cluster. Only relevant when building a mesh of clusters.
+cilium_cluster_name: default
+
+# Make Cilium take ownership over the `/etc/cni/net.d` directory on the node, renaming all non-Cilium CNI configurations to `*.cilium_bak`.
+# This ensures no Pods can be scheduled using other CNI plugins during Cilium agent downtime.
+# Available for Cilium v1.10 and up.
+cilium_cni_exclusive: "true"
+
+# Configure the log file for CNI logging with retention policy of 7 days.
+# Disable CNI file logging by setting this field to empty explicitly.
+# Available for Cilium v1.12 and up.
+cilium_cni_log_file: "/var/run/cilium/cilium-cni.log"

roles/network_plugin/cilium/tasks/apply.yml

@@ -5,7 +5,7 @@
     namespace: "kube-system"
     kubectl: "{{ bin_dir }}/kubectl"
     resource: "{{ item.item.type }}"
-    filename: "{{ kube_config_dir }}/{{ item.item.file }}"
+    filename: "{{ kube_config_dir }}/{{ item.item.name }}-{{ item.item.file }}"
     state: "latest"
   loop: "{{ cilium_node_manifests.results }}"
   when: inventory_hostname == groups['kube_control_plane'][0] and not item is skipped
@@ -25,7 +25,7 @@
     namespace: "kube-system"
     kubectl: "{{ bin_dir }}/kubectl"
     resource: "{{ item.item.type }}"
-    filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.file }}"
+    filename: "{{ kube_config_dir }}/addons/hubble/{{ item.item.name }}-{{ item.item.file }}"
     state: "latest"
   loop: "{{ cilium_hubble_manifests.results }}"
   when:

roles/network_plugin/cilium/tasks/check.yml

@@ -1,14 +1,66 @@
 ---
-- name: Cilium | Check cilium_ipsec_enabled variables
+- name: Cilium | Check Cilium encryption `cilium_ipsec_key` for ipsec
   assert:
     that:
       - "cilium_ipsec_key is defined"
-    msg: "cilium_ipsec_key should be defined to use cilium_ipsec_enabled"
+    msg: "cilium_ipsec_key should be defined to enable encryption using ipsec"
   when:
-    - cilium_ipsec_enabled
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "ipsec"
     - cilium_tunnel_mode in ['vxlan']
 
+# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled`
+- name: Stop if `cilium_ipsec_enabled` is defined and `cilium_encryption_type` is not `ipsec`
+  assert:
+    that: cilium_encryption_type == 'ipsec'
+    msg: >
+      It is not possible to use `cilium_ipsec_enabled` when `cilium_encryption_type` is set to {{ cilium_encryption_type }}.
+  when:
+    - cilium_ipsec_enabled is defined
+    - cilium_ipsec_enabled
+    - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool
+
+- name: Stop if kernel version is too low for Cilium Wireguard encryption
+  assert:
+    that: ansible_kernel.split('-')[0] is version('5.6.0', '>=')
+  when:
+    - kube_network_plugin == 'cilium' or cilium_deploy_additionally | default(false) | bool
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "wireguard"
+    - not ignore_assert_errors
+
 - name: Stop if bad Cilium identity allocation mode
   assert:
     that: cilium_identity_allocation_mode in ['crd', 'kvstore']
     msg: "cilium_identity_allocation_mode must be either 'crd' or 'kvstore'"
+
+- name: Stop if bad Cilium Cluster ID
+  assert:
+    that:
+      - cilium_cluster_id <= 255
+      - cilium_cluster_id >= 0
+    msg: "'cilium_cluster_id' must be between 1 and 255"
+  when: cilium_cluster_id is defined
+
+- name: Stop if bad encryption type
+  assert:
+    that: cilium_encryption_type in ['ipsec', 'wireguard']
+    msg: "cilium_encryption_type must be either 'ipsec' or 'wireguard'"
+  when: cilium_encryption_enabled
+
+- name: Stop if `cilium_encryption_type` is set to "wireguard" and cilium_version is < v1.10.0
+  assert:
+    that: cilium_version | regex_replace('v') is version('1.10', '>')
+    msg: "cilium_encryption_type is set to 'wireguard' but cilium_version is < v1.10.0"
+  when:
+    - cilium_encryption_enabled
+    - cilium_encryption_type == "wireguard"
+
+# TODO: Clean this task up when we drop backward compatibility support for `cilium_ipsec_enabled`
+- name: Set `cilium_encryption_type` to "ipsec" and  if `cilium_ipsec_enabled` is true
+  set_fact:
+    cilium_encryption_type: ipsec
+    cilium_encryption_enabled: true
+  when:
+    - cilium_ipsec_enabled is defined
+    - cilium_ipsec_enabled