Document changes to Kubelet container images

* Build Kubelet container images internally and publish
to Quay and Dockerhub (new) as an alternative in case of
registry outage or breach
* Use our infra to provide single and multi-arch (default)
Kublet images for possible future use
* Docs: Show how to use alternative Kubelet images via
snippets and a systemd dropin (builds on #737)

Changes:

* Update docs with changes to Kubelet image building
* If you prefer to trust images built by Quay/Dockerhub,
automated image builds are still available with unique
tags (albeit with some limitations):
  * Quay automated builds are tagged `build-{short_sha}`
  (limit: only amd64)
  * Dockerhub automated builts are tagged `build-{tag}`
  and `build-master` (limit: only amd64, no shas)

Links:

* Kubelet: https://github.com/poseidon/kubelet
* Docs: https://typhoon.psdn.io/topics/security/#container-images
* Registries:
  * quay.io/poseidon/kubelet
  * docker.io/psdn/kubelet

CHANGES.md

@@ -4,6 +4,14 @@ Notable changes between versions.
 
 ## Latest
 
+* Update Kubelet image build infra and publishing
+  * Publish Kubelet images from internal infra to Quay and Dockerhub
+    * [quay.io/poseidon/kubelet](https://quay.io/repository/poseidon/kubelet) (official)
+    * [docker.io/psdn/kubelet](https://hub.docker.com/r/psdn/kubelet) (fallback)
+  * Document use of alternate Kubelet images during registry incidents
+  * For those preferring to trust images built by Quay/Dockerhub,
+  automated image builds are still available with an alternate tag
+  strategy (see [docs](https://typhoon.psdn.io/topics/security/#container-images))
 * Update Calico from v3.14.0 to [v3.14.1](https://docs.projectcalico.org/v3.14/release-notes/)
 
 ### Addons

docs/advanced/customization.md

@@ -174,3 +174,34 @@ module "nemo" {
 
 To customize low-level Kubernetes control plane bootstrapping, see the [poseidon/terraform-render-bootstrap](https://github.com/poseidon/terraform-render-bootstrap) Terraform module.
 
+## Kubelet
+
+Typhoon publishes Kubelet [container images](/topics/security.md#container-images) to Quay.io (default) and to Dockerhub (in case of a Quay [outage](https://github.com/poseidon/typhoon/issues/735) or breach). Quay automated builds also provide the option for fully verifiable tagged images (`build-{short_sha}`).
+
+To set an alternative Kubelet image, use a snippet to set a systemd dropin.
+
+```
+# host-image-override.yaml
+variant: fcos           <- remove for Flatcar Linux
+version: 1.0.0          <- remove for Flatcar Linux
+systemd:
+  units:
+    - name: kubelet.service
+      dropins:
+        - name: 10-image-override.conf
+          contents: |
+            [Service]
+            Environment=KUBELET_IMAGE=docker.io/psdn/kubelet:v1.18.3
+```
+
+```
+module "nemo" {
+  ...
+
+  worker_snippets = [
+    file("./snippets/host-image-override.yaml")
+  ]
+  ...
+}
+```
+

docs/topics/security.md

@@ -52,7 +52,19 @@ Typhoon uses upstream container images (where possible) and upstream binaries.
 !!! note
     Kubernetes releases `kubelet` as a binary for distros to package, either as a DEB/RPM on traditional distros or as a container image for container-optimized operating systems.
 
-Typhoon [packages](https://github.com/poseidon/kubelet) the upstream Kubelet and its dependencies as a [container image](https://quay.io/repository/poseidon/kubelet) for use in Typhoon. The upstream Kubelet binary is checksummed and packaged directly. Quay automated builds provide verifiability and confidence in image contents.
+Typhoon [packages](https://github.com/poseidon/kubelet) the upstream Kubelet and its dependencies as a [container image](https://quay.io/repository/poseidon/kubelet). Builds fetch the upstream Kubelet binary and verify its checksum.
+
+Typhoon publishes images to Quay.io and Dockerhub.
+
+* [quay.io/poseidon/kubelet](https://quay.io/repository/poseidon/kubelet) (official)
+* [docker.io/psdn/kubelet](https://hub.docker.com/r/psdn/kubelet) (fallback)
+
+Two tag styles indicate the build strategy used.
+
+* Typhoon internal infra publishes single and multi-arch images (e.g. `v1.18.3`, `v1.18.3-amd64`, `v1.18.3-arm64`, `v1.18.3-2-g23228e6-amd64`, `v1.18.3-2-g23228e6-arm64`)
+* Quay and Dockerhub automated builds publish verifiable images (e.g. `build-SHA` on Quay, `build-TAG` on Dockerhub)
+
+The Typhoon-built Kubelet image is used as the official image. Automated builds provide an alternative image for those preferring to trust images built by Quay/Dockerhub (albeit lacking multi-arch). To use the fallback registry or an alternative tag, see [customization](/advanced/customization.md#kubelet).
 
 ## Disclosures