Zephyr CI Infrastructure Architecture v3

[stephanosio]

Overview

This issue describes the "Zephyr CI Infrastructure Architecture v3" and outlines the tasks associated with its implementation.

The goal is to implement a scalable architecture that can expand beyond the current capacity limit of the Centrinix-hosted OpenStack cluster.

The following is a summary of the changes to be made:

• Introduce a new infrastructure-as-a-service (IaaS) provider, Hetzner, who can provide cost effective servers for GitHub Actions CI workloads.
• Set up a Kubernetes cluster on Hetzner bare metal servers.
• Set up GitHub Actions Runner Controller on the Hetzner Kubernetes cluster.
• Shift main GitHub Actions CI workload from the Centrinix cluster to the Hetzner cluster.
• Re-purpose the Centrinix Kubernetes cluster for hardware testing CI (planned for future) and emergency fail-over.

Status quo (v2)

• AWS Kubernetes cluster hosts persistent services, such as Elastic Stack.
• Centrinix Kubernetes cluster hosts ephemeral services, such as GitHub Actions runners.
• AWS Kubernetes cluster may host ephemeral services in case of emergency fail-over.

New (v3)

               +----------------------------+   +-------------------------------------------------------------+
 Services      |    Persistent Workload     |   |                     Ephemeral Workload                      |
               +----------------------------+   +-------------------------------------------------------------+
                                                                                                               
               ------------------------------------------------------------------------------------------------
                                                                                                               
               +----------------------------+   +----------------------------+   +----------------------------+
   PaaS        |     Kubernetes Cluster     |   |     Kubernetes Cluster     |   |     Kubernetes Cluster     |
               +----------------------------+   +----------------------------+   +----------------------------+
                                                                                                               
               ------------------------------------------------------------------------------------------------
                                                                                                               
               +----------------------------+   +----------------------------+   +----------------------------+
               | Elastic Kubernetes Service |   |      OpenStack Magnum      |   |                            |
               +----------------------------+   +----------------------------+   |                            |
   IaaS                                                                          | Hetzner Bare Metal Servers |
               +----------------------------+   +----------------------------+   |                            |
               |  Amazon Web Services EC2   |   |  Centrinix OpenStack Nova  |   |                            |
               +----------------------------+   +----------------------------+   +----------------------------+

• AWS Kubernetes cluster hosts persistent services, such as Elastic Stack.
• Hetzner Kubernetes cluster hosts hardware-agnostic ephemeral services, such as GitHub Actions runners for the workflows that only build tests and/or run tests for emulated platforms.
• Centrinix Kubernetes cluster hosts hardware-dependent ephemeral services, such as GitHub Actions runners for the workflows that build and run tests for non-emulated platforms (i.e. "hardware testing").
• AWS and Centrinix Kubernetes clusters may host hardware-agnostic ephemeral services in case of emergency fail-over.

Specifications

TBD

Tasks

Phase 1 (Initial deployment)

• Deploy Hetzner Kubernetes cluster (initial deployment consisting of 3 server nodes)
• Deploy Actions Runner Controller on Hetzner Kubernetes cluster
• Deploy KeyDB cache on Hetzner Kubernetes cluster

Phase 2 (Scale-up)

TBD