Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ue-rs to image #1148

Merged
merged 2 commits into from
Sep 20, 2023
Merged

Add ue-rs to image #1148

merged 2 commits into from
Sep 20, 2023

Conversation

jepio
Copy link
Member

@jepio jepio commented Sep 18, 2023

Add ue-rs to image

<to be filled by @wrl>

How to use

[ describe what reviewers need to do in order to validate this PR ]

Testing done

[Describe the testing you have done before submitting this PR. Please include both the commands you issued as well as the output you got.]

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

@jepio jepio requested review from wrl and a team September 18, 2023 13:14
@jepio jepio temporarily deployed to development September 18, 2023 13:15 — with GitHub Actions Inactive
krnowak
krnowak reviewed Sep 18, 2023
View reviewed changes
sdk_container/src/third_party/coreos-overlay/coreos-base/ue-rs/ue-rs-0.1.0-r1.ebuild
@@ -0,0 +1 @@
ue-rs-9999.ebuild
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if having ue-rs-9999 makes any sense. Could we have ue-rs-0.1.0-r1.ebuild as a regular file and drop the 9999 one?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could, but the original idea behind the 9999 ebuild is that it makes the history easier to follow (you can blame git log/git blame). if you think this doesn't make sense any more i'll remove the symlink

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would keep this as is

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

git blame should be smart enough to follow renames. But I'm not going to block the PR on this.

@jepio
coreos-base/ue-rs: Initial ebuild
3030926 
Based on the afterburn ebuild with some tweaks. The dependency list was
generated using cargo-ebuild.

Signed-off-by: Jeremi Piotrowski <[email protected]>
@jepio
coreos: Add ue-rs to production image
e259970 
Signed-off-by: Jeremi Piotrowski <[email protected]>
@jepio jepio temporarily deployed to development September 19, 2023 10:27 — with GitHub Actions Inactive
@github-actions
Copy link

Test report for 3728.0.0+nightly-20230915-2100 / amd64 arm64

Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64

ok bpf.execsnoop 🟢 Succeeded: qemu_uefi-amd64 (1)

ok bpf.local-gadget 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.cgroupv1 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.cloudinit.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.cloudinit.multipart-mime 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.cloudinit.script 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.disk.raid0.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.disk.raid0.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.disk.raid1.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.disk.raid1.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.etcd-member.discovery 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.etcd-member.etcdctlv3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.etcd-member.v2-backup-restore 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.filesystem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.flannel.udp 🟢 Succeeded: qemu_uefi-amd64 (1)

ok cl.flannel.vxlan 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.instantiated.enable-unit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.kargs 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.luks 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.indirect 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.indirect.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.regular 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.regular.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.reuse 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.oem.wipe 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.symlink 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.translation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v1.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2_1.ext4checkexisting 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2_1.swap 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.ignition.v2_1.vfat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.install.cloudinit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.internet 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.locksmith.cluster 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.misc.falco 🟢 Succeeded: qemu_uefi-amd64 (1)

ok cl.network.initramfs.second-boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.network.listeners 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.network.wireguard 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.omaha.ping 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.osreset.ignition-rerun 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.overlay.cleanup 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.swap_activation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.sysext.boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.sysext.fallbackdownload # SKIP 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.toolbox.dnf-install 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.update.badverity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.update.grubnop 🟢 Succeeded: qemu_uefi-amd64 (1)

ok cl.update.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.users.shells 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok cl.verity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.auth.verify 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.resource.local 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.resource.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.resource.s3.versioned 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.security.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.ignition.systemd.enable-service 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.locksmith.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.locksmith.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.selinux.boolean 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.selinux.enforce 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.tls.fetch-urls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok coreos.update.badusr 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok devcontainer.docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok devcontainer.systemd-nspawn 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.btrfs-storage 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.containerd-restart 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.lib-coreos-dockerd-compat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.network 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.selinux 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.torcx-manifest-pkgs 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok docker.userns 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.calico.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.cilium.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.25.10.flannel.cgroupv1.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (3) ❌ Failed: qemu_uefi-arm64 (1, 2)

                Diagnostic output for qemu_uefi-arm64, run 2 
    L1: " Error: _cluster.go:117: I0919 14:10:53.631727    1574 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.25"
    L2: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.14"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.14"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.14"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.25.14"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.8"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3"
    L9: "cluster.go:117: I0919 14:11:12.026454    1738 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.25"
    L10: "cluster.go:117: [init] Using Kubernetes version: v1.25.14"
    L11: "cluster.go:117: [preflight] Running pre-flight checks"
    L12: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L13: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L14: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L15: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L16: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L17: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L18: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.5?]"
    L19: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L22: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L27: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L28: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L29: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L30: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L35: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L36: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L37: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L40: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L41: "cluster.go:117: [apiclient] All control plane components are healthy after 6.504172 seconds"
    L42: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L43: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L44: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L45: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L47: "cluster.go:117: [bootstrap-token] Using token: di64j2.0ylrs7vnfwbleckh"
    L48: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L49: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L53: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L54: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L55: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L56: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L57: "cluster.go:117: "
    L58: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L59: "cluster.go:117: "
    L60: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L61: "cluster.go:117: "
    L62: "cluster.go:117:   mkdir -p $HOME/.kube"
    L63: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L64: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L65: "cluster.go:117: "
    L66: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L67: "cluster.go:117: "
    L68: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L69: "cluster.go:117: "
    L70: "cluster.go:117: You should now deploy a pod network to the cluster."
    L71: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L72: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L73: "cluster.go:117: "
    L74: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L75: "cluster.go:117: "
    L76: "cluster.go:117: kubeadm join 10.0.0.5:6443 --token di64j2.0ylrs7vnfwbleckh _"
    L77: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:7bf9acb03c0c2a45408dd05b5e7df2d7f0b5c8d3b81c765c93f2e4dfcc4a0bdd "
    L78: "cluster.go:117: namespace/kube-flannel created"
    L79: "cluster.go:117: clusterrole.rbac.authorization.k8s.io/flannel created"
    L80: "cluster.go:117: clusterrolebinding.rbac.authorization.k8s.io/flannel created"
    L81: "cluster.go:117: serviceaccount/flannel created"
    L82: "cluster.go:117: configmap/kube-flannel-cfg created"
    L83: "cluster.go:117: daemonset.apps/kube-flannel-ds created"
    L84: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L85: "harness.go:582: Found emergency shell on machine 16b30f79-ce12-4b85-921b-c707b3989edb console"
    L86: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine 16b30f79-ce12-4b85-921b-c707b3989edb console"
    L87: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39migni???0m - Ignition (record completion). ) on machine 16b30f79-ce12-4b85-921b-c707b3989edb console_"
    L88: " "
                Diagnostic output for qemu_uefi-arm64, run 1 
    L1: "  "
    L2: " Error: _cluster.go:117: I0919 14:01:45.412567    1552 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.25"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.25.14"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.25.14"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.25.14"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.25.14"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.8"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3"
    L10: "cluster.go:117: I0919 14:01:59.671879    1714 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.25"
    L11: "cluster.go:117: [init] Using Kubernetes version: v1.25.14"
    L12: "cluster.go:117: [preflight] Running pre-flight checks"
    L13: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L14: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L15: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L16: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L17: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L18: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L19: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.6?6]"
    L20: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L22: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L28: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L29: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L30: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L35: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L36: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L37: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L41: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L42: "cluster.go:117: [apiclient] All control plane components are healthy after 7.009901 seconds"
    L43: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L44: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L45: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L47: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L48: "cluster.go:117: [bootstrap-token] Using token: kdj8p9.7lp61oc8goctrgsp"
    L49: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L54: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L55: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L56: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L57: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L58: "cluster.go:117: "
    L59: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L60: "cluster.go:117: "
    L61: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L62: "cluster.go:117: "
    L63: "cluster.go:117:   mkdir -p $HOME/.kube"
    L64: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L65: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L66: "cluster.go:117: "
    L67: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L68: "cluster.go:117: "
    L69: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L70: "cluster.go:117: "
    L71: "cluster.go:117: You should now deploy a pod network to the cluster."
    L72: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L73: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L74: "cluster.go:117: "
    L75: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L76: "cluster.go:117: "
    L77: "cluster.go:117: kubeadm join 10.0.0.66:6443 --token kdj8p9.7lp61oc8goctrgsp _"
    L78: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:b959f33876a36b66d85d63cec62bca6ac38334218f2f80616199341045e5084a "
    L79: "cluster.go:117: namespace/kube-flannel created"
    L80: "cluster.go:117: clusterrole.rbac.authorization.k8s.io/flannel created"
    L81: "cluster.go:117: clusterrolebinding.rbac.authorization.k8s.io/flannel created"
    L82: "cluster.go:117: serviceaccount/flannel created"
    L83: "cluster.go:117: configmap/kube-flannel-cfg created"
    L84: "cluster.go:117: daemonset.apps/kube-flannel-ds created"
    L85: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L86: "harness.go:582: Found emergency shell on machine 66dd9a68-4966-45b8-b50d-da057c6b6294 console"
    L87: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine 66dd9a68-4966-45b8-b50d-da057c6b6294 console"
    L88: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39mtorc???te torcx store to satisfy profile. ) on machine 66dd9a68-4966-45b8-b50d-da057c6b6294 console_"
    L89: " "

ok kubeadm.v1.26.5.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.26.5.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (2) ❌ Failed: qemu_uefi-arm64 (1)

                Diagnostic output for qemu_uefi-arm64, run 1 
    L1: "  "
    L2: " Error: _cluster.go:117: I0919 14:06:38.041307    1445 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.26"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.26.9"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.26.9"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.26.9"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.26.9"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.9"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.6-0"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.9.3"
    L10: "cluster.go:117: I0919 14:06:54.837863    1607 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.26"
    L11: "cluster.go:117: [init] Using Kubernetes version: v1.26.9"
    L12: "cluster.go:117: [preflight] Running pre-flight checks"
    L13: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L14: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L15: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L16: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L17: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L18: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L19: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.1?20]"
    L20: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L21: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L22: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L28: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L29: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L30: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L35: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L36: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L37: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L38: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L41: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L42: "cluster.go:117: [apiclient] All control plane components are healthy after 6.002469 seconds"
    L43: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L44: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L45: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L46: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L47: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L48: "cluster.go:117: [bootstrap-token] Using token: e4c6ic.t91rvdbrxjs1jved"
    L49: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L54: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L55: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L56: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L57: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L58: "cluster.go:117: "
    L59: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L60: "cluster.go:117: "
    L61: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L62: "cluster.go:117: "
    L63: "cluster.go:117:   mkdir -p $HOME/.kube"
    L64: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L65: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L66: "cluster.go:117: "
    L67: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L68: "cluster.go:117: "
    L69: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L70: "cluster.go:117: "
    L71: "cluster.go:117: You should now deploy a pod network to the cluster."
    L72: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L73: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L74: "cluster.go:117: "
    L75: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L76: "cluster.go:117: "
    L77: "cluster.go:117: kubeadm join 10.0.0.120:6443 --token e4c6ic.t91rvdbrxjs1jved _"
    L78: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:61db41579704ca58e2b622d4f47a0f87e526f7f8d530d1229c80a190b4307753 "
    L79: "cluster.go:117: i  Using Cilium version 1.12.5"
    L80: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L81: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L82: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L83: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.5 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L84: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L85: "cluster.go:117: ? Created CA in secret cilium-ca"
    L86: "cluster.go:117: ? Generating certificates for Hubble..."
    L87: "cluster.go:117: ? Creating Service accounts..."
    L88: "cluster.go:117: ? Creating Cluster roles..."
    L89: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.5..."
    L90: "cluster.go:117: i  Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L91: "cluster.go:117: i  Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L92: "cluster.go:117: ? Creating Agent DaemonSet..."
    L93: "cluster.go:117: ? Creating Operator Deployment..."
    L94: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L95: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L96: "cluster.go:117: daemonset.apps/cilium patched"
    L97: "cluster.go:117: ?[33m    /??_"
    L98: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L99: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L100: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L101: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L102: "cluster.go:117: ?[34m    ___/"
    L103: "cluster.go:117: ?[0m"
    L104: "cluster.go:117: Deployment       cilium-operator    "
    L105: "cluster.go:117: DaemonSet        cilium             "
    L106: "cluster.go:117: Containers:      cilium             "
    L107: "cluster.go:117:                  cilium-operator    "
    L108: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L109: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L110: "harness.go:582: Found emergency shell on machine bda53a13-3838-4961-aa35-9f5bc2086327 console"
    L111: "harness.go:582: Found systemd unit failed to start (?[0;1;39mignition-f???es.service?[0m - Ignition (files). ) on machine bda53a13-3838-4961-aa35-9f5bc2086327 console"
    L112: "harness.go:582: Found systemd dependency unit failed to start (?[0;1;39mtorc???te torcx store to satisfy profile. ) on machine bda53a13-3838-4961-aa35-9f5bc2086327 console_"
    L113: " "

ok kubeadm.v1.26.5.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.27.2.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.27.2.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (1) ❌ Failed: qemu_uefi-amd64 (1)

                Diagnostic output for qemu_uefi-amd64, run 1 
    L1: " Error: _cluster.go:117: I0919 13:53:08.193712    1490 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.27"
    L2: "cluster.go:117: W0919 13:53:08.506376    1490 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.6, falling back to the nearest etcd version (3.5.7-0)"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.27.6"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.27.6"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.27.6"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.27.6"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.9"
    L8: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.7-0"
    L9: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1"
    L10: "cluster.go:117: I0919 13:53:19.892519    1654 version.go:256] remote version is much newer: v1.28.2; falling back to: stable-1.27"
    L11: "cluster.go:117: [init] Using Kubernetes version: v1.27.6"
    L12: "cluster.go:117: [preflight] Running pre-flight checks"
    L13: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L14: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L15: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L16: "cluster.go:117: W0919 13:53:20.347923    1654 checks.go:835] detected that the sandbox image __registry.k8s.io/pause:3.8__ of the container runtime is inconsistent with that used by kubeadm. It is rec?ommended that using __registry.k8s.io/pause:3.9__ as the CRI sandbox image."
    L17: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L18: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L19: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L20: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.2?0]"
    L21: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L22: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L23: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L26: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L27: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L28: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L29: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L30: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L31: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L32: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L33: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L34: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L35: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L36: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L37: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L38: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L39: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L40: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L41: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L42: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L43: "cluster.go:117: [apiclient] All control plane components are healthy after 4.002114 seconds"
    L44: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L45: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L46: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L47: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L48: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L49: "cluster.go:117: [bootstrap-token] Using token: g1ur17.9imkzgt1dqcfli85"
    L50: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L52: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L53: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L54: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L55: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L56: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L57: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L58: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L59: "cluster.go:117: "
    L60: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L61: "cluster.go:117: "
    L62: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L63: "cluster.go:117: "
    L64: "cluster.go:117:   mkdir -p $HOME/.kube"
    L65: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L66: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L67: "cluster.go:117: "
    L68: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L69: "cluster.go:117: "
    L70: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L71: "cluster.go:117: "
    L72: "cluster.go:117: You should now deploy a pod network to the cluster."
    L73: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L74: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L75: "cluster.go:117: "
    L76: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L77: "cluster.go:117: "
    L78: "cluster.go:117: kubeadm join 10.0.0.20:6443 --token g1ur17.9imkzgt1dqcfli85 _"
    L79: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:888608fe86f767338f4d9a366cdcd92b3bef635c01ab06545e6f7bfb60cf9128 "
    L80: "cluster.go:117: i  Using Cilium version 1.12.5"
    L81: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L82: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L83: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L84: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.5 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L85: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L86: "cluster.go:117: ? Created CA in secret cilium-ca"
    L87: "cluster.go:117: ? Generating certificates for Hubble..."
    L88: "cluster.go:117: ? Creating Service accounts..."
    L89: "cluster.go:117: ? Creating Cluster roles..."
    L90: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.5..."
    L91: "cluster.go:117: i  Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L92: "cluster.go:117: i  Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L93: "cluster.go:117: ? Creating Agent DaemonSet..."
    L94: "cluster.go:117: ? Creating Operator Deployment..."
    L95: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L96: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L97: "cluster.go:117: daemonset.apps/cilium patched"
    L98: "cluster.go:117: ?[33m    /??_"
    L99: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L100: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L101: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L102: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L103: "cluster.go:117: ?[34m    ___/"
    L104: "cluster.go:117: ?[0m"
    L105: "cluster.go:117: DaemonSet        cilium             "
    L106: "cluster.go:117: Deployment       cilium-operator    "
    L107: "cluster.go:117: Containers:      cilium             "
    L108: "cluster.go:117:                  cilium-operator    "
    L109: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L110: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L111: "--- FAIL: kubeadm.v1.27.2.cilium.base/node_readiness (91.79s)"
    L112: "kubeadm.go:301: nodes are not ready: ready nodes should be equal to 2: 1_"
    L113: " "
    L114: "  "

ok kubeadm.v1.27.2.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.28.1.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok kubeadm.v1.28.1.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (1) ❌ Failed: qemu_uefi-amd64 (1)

                Diagnostic output for qemu_uefi-amd64, run 1 
    L1: " Error: _cluster.go:117: [config/images] Pulled registry.k8s.io/kube-apiserver:v1.28.2"
    L2: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-controller-manager:v1.28.2"
    L3: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-scheduler:v1.28.2"
    L4: "cluster.go:117: [config/images] Pulled registry.k8s.io/kube-proxy:v1.28.2"
    L5: "cluster.go:117: [config/images] Pulled registry.k8s.io/pause:3.9"
    L6: "cluster.go:117: [config/images] Pulled registry.k8s.io/etcd:3.5.9-0"
    L7: "cluster.go:117: [config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1"
    L8: "cluster.go:117: [init] Using Kubernetes version: v1.28.2"
    L9: "cluster.go:117: [preflight] Running pre-flight checks"
    L10: "cluster.go:117: [preflight] Pulling images required for setting up a Kubernetes cluster"
    L11: "cluster.go:117: [preflight] This might take a minute or two, depending on the speed of your internet connection"
    L12: "cluster.go:117: [preflight] You can also perform this action in beforehand using _kubeadm config images pull_"
    L13: "cluster.go:117: W0919 14:02:47.112312    1646 checks.go:835] detected that the sandbox image __registry.k8s.io/pause:3.8__ of the container runtime is inconsistent with that used by kubeadm. It is rec?ommended that using __registry.k8s.io/pause:3.9__ as the CRI sandbox image."
    L14: "cluster.go:117: [certs] Using certificateDir folder __/etc/kubernetes/pki__"
    L15: "cluster.go:117: [certs] Generating __ca__ certificate and key"
    L16: "cluster.go:117: [certs] Generating __apiserver__ certificate and key"
    L17: "cluster.go:117: [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0.0.1?40]"
    L18: "cluster.go:117: [certs] Generating __apiserver-kubelet-client__ certificate and key"
    L19: "cluster.go:117: [certs] Generating __front-proxy-ca__ certificate and key"
    L20: "cluster.go:117: [certs] Generating __front-proxy-client__ certificate and key"
    L21: "cluster.go:117: [certs] External etcd mode: Skipping etcd/ca certificate authority generation"
    L22: "cluster.go:117: [certs] External etcd mode: Skipping etcd/server certificate generation"
    L23: "cluster.go:117: [certs] External etcd mode: Skipping etcd/peer certificate generation"
    L24: "cluster.go:117: [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation"
    L25: "cluster.go:117: [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation"
    L26: "cluster.go:117: [certs] Generating __sa__ key and public key"
    L27: "cluster.go:117: [kubeconfig] Using kubeconfig folder __/etc/kubernetes__"
    L28: "cluster.go:117: [kubeconfig] Writing __admin.conf__ kubeconfig file"
    L29: "cluster.go:117: [kubeconfig] Writing __kubelet.conf__ kubeconfig file"
    L30: "cluster.go:117: [kubeconfig] Writing __controller-manager.conf__ kubeconfig file"
    L31: "cluster.go:117: [kubeconfig] Writing __scheduler.conf__ kubeconfig file"
    L32: "cluster.go:117: [control-plane] Using manifest folder __/etc/kubernetes/manifests__"
    L33: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-apiserver__"
    L34: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-controller-manager__"
    L35: "cluster.go:117: [control-plane] Creating static Pod manifest for __kube-scheduler__"
    L36: "cluster.go:117: [kubelet-start] Writing kubelet environment file with flags to file __/var/lib/kubelet/kubeadm-flags.env__"
    L37: "cluster.go:117: [kubelet-start] Writing kubelet configuration to file __/var/lib/kubelet/config.yaml__"
    L38: "cluster.go:117: [kubelet-start] Starting the kubelet"
    L39: "cluster.go:117: [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory __/etc/kubernetes/manifests__. This can take up to 30m0s"
    L40: "cluster.go:117: [apiclient] All control plane components are healthy after 4.502683 seconds"
    L41: "cluster.go:117: [upload-config] Storing the configuration used in ConfigMap __kubeadm-config__ in the __kube-system__ Namespace"
    L42: "cluster.go:117: [kubelet] Creating a ConfigMap __kubelet-config__ in namespace kube-system with the configuration for the kubelets in the cluster"
    L43: "cluster.go:117: [upload-certs] Skipping phase. Please see --upload-certs"
    L44: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]"
    L45: "cluster.go:117: [mark-control-plane] Marking the node localhost as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]"
    L46: "cluster.go:117: [bootstrap-token] Using token: cvqsus.ua893lr7inr9l0f0"
    L47: "cluster.go:117: [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles"
    L48: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes"
    L49: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials"
    L50: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token"
    L51: "cluster.go:117: [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster"
    L52: "cluster.go:117: [bootstrap-token] Creating the __cluster-info__ ConfigMap in the __kube-public__ namespace"
    L53: "cluster.go:117: [kubelet-finalize] Updating __/etc/kubernetes/kubelet.conf__ to point to a rotatable kubelet client certificate and key"
    L54: "cluster.go:117: [addons] Applied essential addon: CoreDNS"
    L55: "cluster.go:117: [addons] Applied essential addon: kube-proxy"
    L56: "cluster.go:117: "
    L57: "cluster.go:117: Your Kubernetes control-plane has initialized successfully!"
    L58: "cluster.go:117: "
    L59: "cluster.go:117: To start using your cluster, you need to run the following as a regular user:"
    L60: "cluster.go:117: "
    L61: "cluster.go:117:   mkdir -p $HOME/.kube"
    L62: "cluster.go:117:   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config"
    L63: "cluster.go:117:   sudo chown $(id -u):$(id -g) $HOME/.kube/config"
    L64: "cluster.go:117: "
    L65: "cluster.go:117: Alternatively, if you are the root user, you can run:"
    L66: "cluster.go:117: "
    L67: "cluster.go:117:   export KUBECONFIG=/etc/kubernetes/admin.conf"
    L68: "cluster.go:117: "
    L69: "cluster.go:117: You should now deploy a pod network to the cluster."
    L70: "cluster.go:117: Run __kubectl apply -f [podnetwork].yaml__ with one of the options listed at:"
    L71: "cluster.go:117:   https://kubernetes.io/docs/concepts/cluster-administration/addons/"
    L72: "cluster.go:117: "
    L73: "cluster.go:117: Then you can join any number of worker nodes by running the following on each as root:"
    L74: "cluster.go:117: "
    L75: "cluster.go:117: kubeadm join 10.0.0.140:6443 --token cvqsus.ua893lr7inr9l0f0 _"
    L76: "cluster.go:117:  --discovery-token-ca-cert-hash sha256:629b61a25d980375900c34c3f8ae12d1d1ac6560ef9dc34f0abe4fdeaccc09b2 "
    L77: "cluster.go:117: i  Using Cilium version 1.12.5"
    L78: "cluster.go:117: ? Auto-detected cluster name: kubernetes"
    L79: "cluster.go:117: ? Auto-detected datapath mode: tunnel"
    L80: "cluster.go:117: ? Auto-detected kube-proxy has been installed"
    L81: "cluster.go:117: i  helm template --namespace kube-system cilium cilium/cilium --version 1.12.5 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,extraConfig.cluster-pool-ipv4-?cidr=192.168.0.0/17,extraConfig.enable-endpoint-routes=true,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vx?lan"
    L82: "cluster.go:117: i  Storing helm values file in kube-system/cilium-cli-helm-values Secret"
    L83: "cluster.go:117: ? Created CA in secret cilium-ca"
    L84: "cluster.go:117: ? Generating certificates for Hubble..."
    L85: "cluster.go:117: ? Creating Service accounts..."
    L86: "cluster.go:117: ? Creating Cluster roles..."
    L87: "cluster.go:117: ? Creating ConfigMap for Cilium version 1.12.5..."
    L88: "cluster.go:117: i  Manual overwrite in ConfigMap: enable-endpoint-routes=true"
    L89: "cluster.go:117: i  Manual overwrite in ConfigMap: cluster-pool-ipv4-cidr=192.168.0.0/17"
    L90: "cluster.go:117: ? Creating Agent DaemonSet..."
    L91: "cluster.go:117: ? Creating Operator Deployment..."
    L92: "cluster.go:117: ? Waiting for Cilium to be installed and ready..."
    L93: "cluster.go:117: ? Cilium was successfully installed! Run _cilium status_ to view installation health"
    L94: "cluster.go:117: daemonset.apps/cilium patched"
    L95: "cluster.go:117: ?[33m    /??_"
    L96: "cluster.go:117: ?[36m /???[33m___/?[32m??_?[0m    Cilium:         ?[32mOK?[0m"
    L97: "cluster.go:117: ?[36m ___?[31m/??_?[32m__/?[0m    Operator:       ?[32mOK?[0m"
    L98: "cluster.go:117: ?[32m /???[31m___/?[35m??_?[0m    Hubble:         ?[36mdisabled?[0m"
    L99: "cluster.go:117: ?[32m ___?[34m/??_?[35m__/?[0m    ClusterMesh:    ?[36mdisabled?[0m"
    L100: "cluster.go:117: ?[34m    ___/"
    L101: "cluster.go:117: ?[0m"
    L102: "cluster.go:117: Deployment       cilium-operator    "
    L103: "cluster.go:117: DaemonSet        cilium             "
    L104: "cluster.go:117: Containers:      cilium             "
    L105: "cluster.go:117:                  cilium-operator    "
    L106: "cluster.go:117: Cluster Pods:    0/0 managed by Cilium"
    L107: "cluster.go:117: Created symlink /etc/systemd/system/multi-user.target.wants/kubelet.service ??? /etc/systemd/system/kubelet.service."
    L108: "--- FAIL: kubeadm.v1.28.1.cilium.base/node_readiness (91.95s)"
    L109: "kubeadm.go:301: nodes are not ready: ready nodes should be equal to 2: 1_"
    L110: " "
    L111: "  "

ok kubeadm.v1.28.1.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok linux.nfs.v3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok linux.nfs.v4 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok linux.ntp 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok packages 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok skipped qemu update tests 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

ok systemd.journal.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok systemd.journal.user 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok systemd.sysext.custom-docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok systemd.sysext.custom-oem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok systemd.sysext.simple 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok systemd.sysusers.gshadow 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

ok torcx.enable-service 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

t-lo
t-lo approved these changes Sep 19, 2023
View reviewed changes
Copy link
Member

@t-lo t-lo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ebuild looks sensible and tests run fine. LGTM.

@jepio jepio merged commit bc8261f into main Sep 20, 2023
@jepio jepio deleted the add-ue-rs branch September 20, 2023 08:53
@github-actions github-actions bot mentioned this pull request Nov 9, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@t-lo t-lo t-lo approved these changes

@krnowak krnowak krnowak approved these changes

@wrl wrl Awaiting requested review from wrl

Assignees
No one assigned
Labels
main
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jepio @t-lo @krnowak @dongsupark