Welcome to my opinionated and extensible template for deploying a single Kubernetes cluster. The goal of this project is to make it easier for people interested in using Kubernetes to deploy a cluster at home on bare-metal or VMs. This template closely mirrors my personal home-ops repository.

At a high level this project makes use of makejinja to read in configuration files (cluster.yaml & nodes.yaml). Makejinja will render out templates that will allow you to install a Kubernetes cluster with the features mentioned below.

A Kubernetes cluster deployed on-top of Talos Linux with an opinionated implementation of Flux using GitHub as the Git provider, sops to manage secrets and cloudflared to access applications external to your local network.

• Required: Some knowledge of Containers, YAML, Git, and a Cloudflare account with a domain.
• Included components: flux, cilium, cert-manager, spegel, reloader, ingress-nginx, external-dns and cloudflared.

Other features include:

• Dev env managed w/ mise
• Workflow automation w/ GitHub Actions
• Dependency automation w/ Renovate
• Flux HelmRelease and Kustomization diffs w/ flux-local

Does this sound cool to you? If so, continue to read on.

There are 5 stages outlined below for completing this project, make sure you follow the stages in order.

1. Head over to the Talos Linux Image Factory and follow the instructions. Be sure to only choose the bare-minimum system extensions as some might require additional configuration and prevent Talos from booting without it. You can always add system extensions after Talos is installed and working.

2. This will eventually lead you to download a Talos Linux ISO (or for SBCs a RAW) image. Make sure to note the schematic ID you will need this later on.

3. Flash the Talos ISO or RAW image to a USB drive and boot from it on your nodes.

4. Verify with nmap that your nodes are available on the network. (Replace 192.168.1.0/24 with the network your nodes are on.)

   nmap -Pn -n -p 50000 192.168.1.0/24 -vv | grep 'Discovered'

1. Create a new repository by clicking the green Use this template button at the top of this page, then clone the new repo you just created and cd into it. Alternatively you can us the GitHub CLI ...

   export REPONAME="home-ops"
   gh repo create $REPONAME --template onedr0p/cluster-template --disable-wiki --public --clone && cd $REPONAME

2. Install and activate mise following the instructions for your workstation here.

3. Use mise to install the required CLI tools:

   📍 If mise is having trouble compiling Python, try running mise settings python.compile=0 and try these commands again

   mise trust && mise install && mise run deps

1. Create a Cloudflare API token for use with cloudflared and external-dns by reviewing the official documentation and following the instructions below.

   • Click the blue Use template button for the Edit zone DNS template.
   • Name your token kubernetes
   • Under Permissions, click + Add More and add permissions Zone - DNS - Edit and Account - Cloudflare Tunnel - Read
   • Limit the permissions to a specific account and/or zone resources and then click Continue to Summary and then Create Token.
   • Save this token somewhere safe, you will need it later on.

2. Create the Cloudflare Tunnel:

   cloudflared tunnel login
   cloudflared tunnel create --credentials-file cloudflare-tunnel.json kubernetes

1. Generate the config files from the sample files:

   task init

2. Fill out cluster.yaml and nodes.yaml configuration files using the comments in those file as a guide.

3. Template out the kubernetes and talos configuration files, if any issues come up be sure to read the error and adjust your config files accordingly.

   task configure

4. Push your changes to git:

   📍 Verify all the ./kubernetes/**/*.sops.* files are encrypted with SOPS

   git add -A
   git commit -m "chore: initial commit :rocket:"
   git push

1. Install Talos:

   task bootstrap:talos

2. Push your changes to git:

   git add -A
   git commit -m "chore: add talhelper encrypted secret :lock:"
   git push

3. Install cilium, coredns, spegel, flux and sync the cluster to the repository state:

   task bootstrap:apps

4. Watch the rollout of your cluster happen:

   kubectl get pods --all-namespaces --watch

The external-dns application created in the networking namespace will handle creating public DNS records. By default, echo-server and the flux-webhook are the only subdomains reachable from the public internet. In order to make additional applications public you must set set the correct ingress class name and ingress annotations like in the HelmRelease for echo-server.

k8s_gateway will provide DNS resolution to external Kubernetes resources (i.e. points of entry to the cluster) from any device that uses your home DNS server. For this to work, your home DNS server must be configured to forward DNS queries for ${cloudflare_domain} to ${cloudflare_dns_gateway_addr} instead of the upstream DNS server(s) it normally uses. This is a form of split DNS (aka split-horizon DNS / conditional forwarding).

... Nothing working? That is expected, this is DNS after all!

Steps to update to the Let's Encrypt production environment:

1. In cluster.yaml update cloudflare_cluster_issuer to production

2. Run task configure

3. Push your changes to git:

   git add -A
   git commit -m "chore: switch to le-prod :scroll:"
   git push

4. Wait for your certificate to be created, you can check the status by running:

   kubectl -n cert-manager describe certificate <name>

By default Flux will periodically check your git repository for changes. In-order to have Flux reconcile on git push you must configure Github to send push events to Flux.

1. Obtain the webhook path:

   📍 Hook id and path should look like /hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123

   kubectl -n flux-system get receiver github-webhook --output=jsonpath='{.status.webhookPath}'

2. Piece together the full URL with the webhook path appended:

   https://flux-webhook.${cloudflare_domain}/hook/12ebd1e363c641dc3c2e430ecf3cee2b3c7a5ac9e1234506f6f5f3ce1230e123
   

3. Navigate to the settings of your repository on Github, under "Settings/Webhooks" press the "Add webhook" button. Fill in the webhook URL and your token from github-push-token.txt, Content type: application/json, Events: Choose Just the push event, and save.

There might be a situation where you want to destroy your Kubernetes cluster. The following command will reset your nodes back to maintenance mode.

task talos:reset

# (Re)generate the Talos config
task talos:generate-config
# Apply the config to the node
task talos:apply-node IP=? MODE=?
# e.g. task talos:apply-node IP=10.10.10.10 MODE=auto

# Upgrade node to a newer Talos version
task talos:upgrade-node IP=?
# e.g. task talos:upgrade-node IP=10.10.10.10

# Upgrade cluster to a newer Kubernetes version
task talos:upgrade-k8s
# e.g. task talos:upgrade-k8s

Renovate is a tool that automates dependency management. It is designed to scan your repository around the clock and open PRs for out-of-date dependencies it finds. Common dependencies it can discover are Helm charts, container images, GitHub Actions, Ansible roles... even Flux itself! Merging a PR will cause Flux to apply the update to your cluster.

To enable Renovate, click the 'Configure' button over at their Github app page and select your repository. Renovate creates a "Dependency Dashboard" as an issue in your repository, giving an overview of the status of all updates. The dashboard has interactive checkboxes that let you do things like advance scheduling or reattempt update PRs you closed without merging.

The base Renovate configuration in your repository can be viewed at .github/renovate.json5. By default it is scheduled to be active with PRs every weekend, but you can change the schedule to anything you want, or remove it if you want Renovate to open PRs right away.

Below is a general guide on trying to debug an issue with an resource or application. For example, if a workload/resource is not showing up or a pod has started but in a CrashLoopBackOff or Pending state. Most of these steps do not include a way to fix the problem as the problem could be one of many different things.

1. Verify the Git Repository is up-to-date and in a ready state.

   flux get sources git -A

   Force Flux to sync your repository to your cluster:

   flux -n flux-system reconcile ks flux-system --with-source

2. Verify all the Flux kustomizations are up-to-date and in a ready state.

   flux get ks -A

3. Verify all the Flux helm releases are up-to-date and in a ready state.

   flux get hr -A

4. Do you see the pod of the workload you are debugging?

   kubectl -n <namespace> get pods -o wide

5. Check the logs of the pod if its there.

   kubectl -n <namespace> logs <pod-name> -f

6. If a resource exists try to describe it to see what problems it might have.

   kubectl -n <namespace> describe <resource> <name>

7. Check the namespace events

   kubectl -n <namespace> get events --sort-by='.metadata.creationTimestamp'

Resolving problems that you have could take some tweaking of your YAML manifests in order to get things working, other times it could be a external factor like permissions on a NFS server. If you are unable to figure out your problem see the support sections below.

Once your cluster is fully configured and you no longer need to run task configure, it's a good idea to clean up the repository by removing the templates directory and any files related to the templating process. This will help eliminate unnecessary clutter from the upstream template repository and resolve any "duplicate registry" warnings from Renovate.

1. Tidy up your repository:

   task template:tidy

2. Push your changes to git:

   git add -A
   git commit -m "chore: tidy up :broom:"
   git push

• Make a post in this repository's Github Discussions.
• Start a thread in the #support or #cluster-template channels in the Home Operations Discord server.

If you're having difficulty with this project, can't find the answers you need through the community support options above, or simply want to show your appreciation while gaining deeper insights, I’m offering one-on-one paid support through GitHub Sponsors for a limited time. Payment and scheduling will be coordinated through GitHub Sponsors.

The cluster is your oyster (or something like that). Below are some optional considerations you might want to review.

Instead of using k8s_gateway to provide DNS for your applications you might want to check out external-dns, it has wide support for many different providers such as Pi-hole, UniFi, Adguard Home, Bind and more.

You might find you need persistent storage for your workloads with features like replicated storage or to connect to a NFS/SMB/iSCSI server. If you need any of those features be sure to check out the projects like rook-ceph, longhorn, openebs, democratic-csi, csi-driver-nfs, csi-driver-smb or synology-csi.

Community member @whazor created Kubesearch to allow searching Flux HelmReleases across Github and Gitlab repositories with the kubesearch topic.

If this repo is too hot to handle or too cold to hold check out these following projects.

• ajaykumar4/cluster-template - A template for deploying a Talos Kubernetes cluster including Argo for GitOps
• khuedoan/homelab - Fully automated homelab from empty disk to running services with a single command.
• ricsanfre/pi-cluster - Pi Kubernetes Cluster. Homelab kubernetes cluster automated with Ansible and FluxCD
• techno-tim/k3s-ansible - The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat.

Big shout out to all the contributors, sponsors and everyone else who has helped on this project.