salt: Allow to provide featureGates for apiserver in bootstrap config

Add ability to provide some feature gates for kube apiserver in
bootstrap config that will be persisted after upgrade/salt highstate ...

Fixes: #3294

CHANGELOG.md

@@ -6,6 +6,10 @@
   Alertmanager are now stored in Loki database for persistence
   (PR[#3191](https://github.com/scality/metalk8s/pull/3191))
 
+- [#3294](https://github.com/scality/metalk8s/issues/3294) - Allow to manage
+  `kube-apiserver` feature gates from Bootstrap Configuration file
+  (PR[#3318](https://github.com/scality/metalk8s/pull/3318))
+
 ### Enhancements
 - Bump Kubernetes version to 1.20.6
   (PR[#3311](https://github.com/scality/metalk8s/pull/3311))

docs/installation/bootstrap.rst

@@ -62,6 +62,10 @@ Configuration
         minion: <hostname-of-the-bootstrap-node>
       archives:
         - <path-to-metalk8s-iso>
+      kubernetes:
+        apiServer:
+          featureGates:
+            <feature_gate_name>: True
 
 The ``networks`` field specifies a range of IP addresses written in CIDR
 notation for it's various subfields.
@@ -130,6 +134,13 @@ The ``archives`` field is a list of absolute paths to MetalK8s ISO files. When
 the bootstrap script is executed, those ISOs are automatically mounted and the
 system is configured to re-mount them automatically after a reboot.
 
+The ``kubernetes`` field can be omitted if you do not have any specific
+Kubernetes `Feature Gates`_ to enable or disable.
+If you need to enable or disable a specific features for ``kube-apiserver``
+configure the corresponding entries in the ``kubernetes.apiServer.featureGates``
+mapping.
+
+.. _Feature Gates: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
 
 .. _Bootstrap SSH Provisioning:
 

salt/_pillar/metalk8s.py

@@ -170,6 +170,7 @@ def ext_pillar(minion_id, pillar, bootstrap_config):  # pylint: disable=unused-a
             "networks": _load_networks(config),
             "metalk8s": metal_data,
             "proxies": config.get("proxies", {}),
+            "kubernetes": config.get("kubernetes", {}),
         }
 
         if not isinstance(metal_data["archives"], list):

salt/metalk8s/kubernetes/apiserver/installed.sls

@@ -26,6 +26,16 @@ include:
 {%- endif %}
 {%- set etcd_servers = etcd_servers | unique %}
 
+{%- set feature_gates = [] %}
+{%- for feature, value in pillar.kubernetes.get("apiServer", {}).get("featureGates", {}).items() %}
+{%-   if value is True %}
+{%-     set value = "true" %}
+{%-   elif value is False %}
+{%-     set value = "false" %}
+{%-   endif %}
+{%-   do feature_gates.append(feature ~ "=" ~ value) %}
+{%- endfor %}
+
 Create kube-apiserver Pod manifest:
   metalk8s.static_pod_managed:
     - name: /etc/kubernetes/manifests/kube-apiserver.yaml
@@ -96,6 +106,9 @@ Create kube-apiserver Pod manifest:
           - --oidc-groups-claim=groups
           - '"--oidc-groups-prefix=oidc:"'
           - --v={{ 2 if metalk8s.debug else 0 }}
+          {% if feature_gates %}
+          - --feature-gates={{ feature_gates | join(",") }}
+          {%- endif %}
         requested_cpu: 250m
         volumes:
           - path: {{ encryption_k8s_path }}

salt/tests/unit/formulas/data/base_pillar.yaml

@@ -172,3 +172,4 @@ certificates:
         watched: true
       workload-plane-ingress:
         watched: true
+kubernetes: {}