Add support for building aarch64 ISO #13762

sharifelgamal · 2022-03-09T16:31:18Z

This rather large PR adds the ability to build an arm64 buildroot ISO.
Specific changes include:

adding CHANGELOG to the ISO rootfs overlay
support for building ISO with UEFI bootloader (although the amd64 ISO remains BIOS for several reasons)
full refactor of the deploy/iso/minikube-iso subdirectories
- adding the arch directory for all arch specific packages
- templatizing Config.in so each arch can define it at build time
- move board/coreos/minikube to board/minikube/<arch>
Make sure automation builds and recognizes both ISOs
Make sure minikube auto-selects ISO based on arch automatically

Closes #9228

Thanks for your patience everyone.

pkg/minikube/node/start.go

.github/workflows/update-k8s-versions.yml

Makefile

deploy/iso/minikube-iso/arch/aarch64/package/cri-dockerd-aarch64/cri-dockerd.mk

sharifelgamal · 2022-05-06T22:52:53Z

deep breath here we go

/ok-to-test

minikube-pr-bot · 2022-05-06T23:42:12Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 50.8s    | 51.1s               |
| enable ingress | 28.7s    | 29.7s               |
+----------------+----------+---------------------+

Times for minikube start: 52.0s 50.5s 50.6s 49.7s 51.3s
Times for minikube (PR 13762) start: 51.8s 51.2s 50.8s 50.8s 51.1s

Times for minikube ingress: 26.0s 30.1s 30.0s 29.0s 28.6s
Times for minikube (PR 13762) ingress: 28.6s 30.1s 30.1s 29.5s 30.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 24.6s    | 24.7s               |
| enable ingress | 23.1s    | 23.0s               |
+----------------+----------+---------------------+

Times for minikube start: 26.0s 24.7s 23.9s 24.5s 23.7s
Times for minikube (PR 13762) start: 24.5s 24.4s 24.1s 25.7s 25.0s

Times for minikube ingress: 24.4s 22.9s 23.9s 22.4s 21.9s
Times for minikube (PR 13762) ingress: 22.9s 23.9s 23.9s 21.9s 22.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 39.3s    | 39.8s               |
| enable ingress | 20.8s    | 21.7s               |
+----------------+----------+---------------------+

Times for minikube start: 32.9s 40.3s 43.9s 39.8s 39.7s
Times for minikube (PR 13762) start: 39.2s 39.8s 43.7s 32.7s 43.7s

Times for minikube ingress: 19.0s 22.4s 17.9s 22.4s 22.4s
Times for minikube (PR 13762) ingress: 31.9s 22.4s 18.4s 17.9s 17.9s

medyagh · 2022-05-07T00:33:12Z

omg ! Am I dreaming ? KVM and Hyperkit Tests are Green on this ISO PR ! this is one of the biggest works delivered in minikube in years ! Thank you for taking this to the finish line

sharifelgamal · 2022-05-07T01:28:01Z

ok-to-build-iso

minikube-pr-bot · 2022-05-07T01:41:04Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_macOS	TestFunctional/parallel/ImageCommands/ImageListTable (gopogh)	1.06 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/auto/DNS (gopogh)	17.33 (chart)
Docker_Linux_containerd	TestAddons/Setup (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestIngressAddonLegacy/StartLegacyK8sCluster (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestJSONOutput/start/Audit (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestJSONOutput/start/Command (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/AddNode (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/DeleteNode (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/FreshStart2Nodes (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/ProfileList (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/RestartKeepsNodes (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/RestartMultiNode (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/StartAfterStop (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestMultiNode/serial/StopNode (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestPreload (gopogh)	24.76 (chart)
Docker_Linux_containerd	TestOffline (gopogh)	27.62 (chart)
Docker_Linux_containerd	TestPause/serial/Start (gopogh)	27.62 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MountCmd/any-port (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MySQL (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/PersistentVolumeClaim (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmd (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmdConnect (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/serial/ExtraConfig (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/serial/SoftStart (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestFunctional/serial/StartWithProxy (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestMultiNode/serial/DeployApp2Nodes (gopogh)	38.10 (chart)
Docker_Linux_containerd	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	38.10 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

minikube-pr-bot · 2022-05-07T02:17:14Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 51.9s    | 51.0s               |
| enable ingress | 28.0s    | 29.0s               |
+----------------+----------+---------------------+

Times for minikube start: 53.8s 51.4s 51.7s 51.2s 51.3s
Times for minikube (PR 13762) start: 51.0s 51.4s 51.0s 50.0s 51.4s

Times for minikube ingress: 25.1s 29.6s 30.1s 29.1s 26.1s
Times for minikube (PR 13762) ingress: 31.5s 30.5s 28.6s 28.5s 26.0s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 25.0s    | 24.0s               |
| enable ingress | 22.6s    | 23.7s               |
+----------------+----------+---------------------+

Times for minikube start: 27.9s 24.6s 23.7s 24.6s 24.2s
Times for minikube (PR 13762) start: 24.1s 23.9s 24.3s 23.4s 24.1s

Times for minikube ingress: 22.9s 24.4s 20.9s 21.9s 22.9s
Times for minikube (PR 13762) ingress: 22.4s 26.9s 22.9s 24.0s 22.4s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 39.5s    | 38.5s               |
| enable ingress | 22.8s    | 27.3s               |
+----------------+----------+---------------------+

Times for minikube (PR 13762) start: 39.9s 40.0s 29.3s 39.8s 43.5s
Times for minikube start: 29.1s 38.8s 40.2s 44.8s 44.4s

Times for minikube ingress: 22.4s 22.4s 32.5s 18.4s 18.4s
Times for minikube (PR 13762) ingress: 21.9s 21.9s 22.4s 51.9s 18.4s

minikube-bot · 2022-05-07T02:53:08Z

Hi @sharifelgamal, building a new ISO failed.
See the logs at: https://storage.cloud.google.com/minikube-builds/logs/13762/66a8e87/iso_build.txt

sharifelgamal · 2022-05-07T03:02:48Z

ok-to-build-iso

minikube-pr-bot · 2022-05-07T03:59:37Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux	TestNoKubernetes/serial/StartNoArgs (gopogh)	3.26 (chart)
Docker_Linux_containerd	TestPause/serial/Pause (gopogh)	10.39 (chart)
Docker_macOS	TestStartStop/group/default-k8s-different-port/serial/AddonExistsAfterStop (gopogh)	44.68 (chart)
Docker_Windows	TestForceSystemdFlag (gopogh)	52.53 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/kindnet/Start (gopogh)	54.05 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/calico/DNS (gopogh)	63.64 (chart)
Docker_macOS	TestNetworkPlugins/group/calico/Start (gopogh)	65.79 (chart)
Docker_macOS	TestNetworkPlugins/group/kubenet/DNS (gopogh)	66.67 (chart)
Docker_Windows	TestStartStop/group/newest-cni/serial/Pause (gopogh)	67.35 (chart)
Docker_macOS	TestNetworkPlugins/group/kindnet/Start (gopogh)	69.74 (chart)
Docker_macOS	TestNetworkPlugins/group/bridge/DNS (gopogh)	73.91 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/enable-default-cni/DNS (gopogh)	75.64 (chart)
Docker_macOS	TestNetworkPlugins/group/enable-default-cni/DNS (gopogh)	77.36 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/bridge/DNS (gopogh)	78.48 (chart)
Docker_macOS	TestDownloadOnly/v1.16.0/preload-exists (gopogh)	98.98 (chart)
Docker_Windows	TestFunctional/parallel/ServiceCmd (gopogh)	100.00 (chart)
Docker_Windows	TestNetworkPlugins/group/cilium/Start (gopogh)	100.00 (chart)
Docker_Windows	TestSkaffold (gopogh)	100.00 (chart)

To see the flake rates of all tests by environment, click here.

minikube-bot · 2022-05-07T05:18:13Z

Hi @sharifelgamal, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

minikube-pr-bot · 2022-05-07T06:07:14Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 51.3s    | 52.1s               |
| enable ingress | 29.4s    | 28.5s               |
+----------------+----------+---------------------+

Times for minikube start: 53.5s 50.7s 51.2s 49.7s 51.3s
Times for minikube (PR 13762) start: 51.1s 51.4s 51.7s 54.0s 52.4s

Times for minikube ingress: 30.5s 29.1s 30.0s 29.6s 28.1s
Times for minikube (PR 13762) ingress: 30.0s 28.5s 29.6s 27.1s 27.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 25.2s    | 23.7s               |
| enable ingress | 22.8s    | 22.8s               |
+----------------+----------+---------------------+

Times for minikube ingress: 23.9s 22.4s 22.4s 22.9s 22.4s
Times for minikube (PR 13762) ingress: 21.9s 22.9s 23.4s 22.9s 23.0s

Times for minikube start: 28.8s 24.6s 24.5s 24.0s 24.1s
Times for minikube (PR 13762) start: 23.5s 23.9s 23.9s 23.7s 23.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13762) |
+----------------+----------+---------------------+
| minikube start | 39.2s    | 36.8s               |
| enable ingress | 20.6s    | 23.0s               |
+----------------+----------+---------------------+

Times for minikube start: 32.5s 43.9s 40.4s 39.8s 39.4s
Times for minikube (PR 13762) start: 39.9s 32.0s 39.4s 28.8s 43.8s

Times for minikube ingress: 17.9s 17.9s 22.4s 22.4s 22.5s
Times for minikube (PR 13762) ingress: 22.4s 28.9s 23.4s 22.4s 17.9s

minikube-pr-bot · 2022-05-07T07:37:58Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux	TestNoKubernetes/serial/Start (gopogh)	1.05 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/auto/Start (gopogh)	5.26 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/cilium/Start (gopogh)	6.58 (chart)
Hyperkit_macOS	TestFunctional/serial/ComponentHealth (gopogh)	8.33 (chart)
Docker_Linux_containerd	TestAddons/parallel/Registry (gopogh)	17.28 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/flannel/Start (gopogh)	21.59 (chart)
Docker_Linux_containerd	TestNetworkPlugins/group/custom-weave/Start (gopogh)	22.37 (chart)
Docker_Linux_containerd	TestOffline (gopogh)	27.78 (chart)
Docker_Linux	TestFunctional/serial/ComponentHealth (gopogh)	35.79 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/bridge/Start (gopogh)	36.36 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MountCmd/any-port (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/MySQL (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/PersistentVolumeClaim (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmd (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/ServiceCmdConnect (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/serial/ExtraConfig (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/serial/SoftStart (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/serial/StartWithProxy (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestMultiNode/serial/DeployApp2Nodes (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	37.96 (chart)
Docker_Linux_containerd	TestFunctional/parallel/DashboardCmd (gopogh)	38.89 (chart)
Docker_Linux	TestNetworkPlugins/group/false/DNS (gopogh)	40.00 (chart)
Hyper-V_Windows	TestNetworkPlugins/group/kubenet/Start (gopogh)	44.32 (chart)
Docker_Windows	TestAddons/Setup (gopogh)	49.50 (chart)
Docker_Windows	TestFunctional/parallel/CertSync (gopogh)	49.50 (chart)
Docker_Windows	TestFunctional/parallel/CpCmd (gopogh)	49.50 (chart)
Docker_Windows	TestFunctional/parallel/DockerEnv/powershell (gopogh)	49.50 (chart)
Docker_Windows	TestFunctional/parallel/FileSync (gopogh)	49.50 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

afbjorklund · 2022-05-07T10:22:40Z

I am not a big fan of duplicating the package directories, but I would also be happy to refactor this after it has been delivered...

There should only be one package directory (for each "external" project), and there is no need to add a template to Config.in

Each package already has a dependency on the arch, where needed.

depends on BR2_x86_64
depends on BR2_AARCH64

sharifelgamal · 2022-05-07T17:03:46Z

I am not a big fan of duplicating the package directories, but I would also be happy to refactor this after it has been delivered...

There should only be one package directory (for each "external" project), and there is no need to add a template to Config.in

Each package already has a dependency on the arch, where needed.

depends on BR2_x86_64 depends on BR2_AARCH64

yeah, this was my understanding as well, but i was having issues with arch's building the wrong packages, so i built this system to make sure it didn't happen while testing

afbjorklund · 2022-05-07T17:45:20Z

i built this system to make sure it didn't happen while testing

sounds reasonable, it can be changed later without affecting the pacakges... i.e. the end make is the same, either way.

the main annoyance was when having to translate from the OS arch to the Go arch, but that's more tedious than "hard"

k8s-ci-robot · 2022-05-09T22:55:48Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, sharifelgamal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh,sharifelgamal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

klaases and others added 16 commits February 14, 2022 14:19

build iso with buildroot defaults

ef92c0f

Increase BR2_TARGET_ROOTFS_EXT2_SIZE

8b42dc2

Add some logging

9370f01

Disable BR2_TARGET_ROOTFS_EXT2

0135dbf

Working UEFI config

296920b

Update KVM2 configuration for UEFI

1be7018

Dev arm defconfig

02461d3

Updating ISO to v1.25.0-1645046644-13612

276d728

Remove commented defconfig settings

e2aa1dd

Fix bzImage path

03b064d

Match capitalization to driver.go entry

d891262

Remove .img from initrd

9786d0a

Merge branch 'uefi2' of github.com:klaases/minikube into iso-arm64

5f29ddf

Merge branch 'narm1' of github.com:klaases/minikube into iso-arm64

2e3946f

name stuff arm64

0f63390

fix config name reference

1b83267

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 9, 2022

k8s-ci-robot requested review from afbjorklund and blueelvis March 9, 2022 16:32

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 9, 2022

sharifelgamal added 8 commits March 9, 2022 15:24

shuffling many things around

325ab07

remove generated files

bca48ce

delete old iso board location

782708e

gendocs

30472c9

fix iso location

4123c12

fix make target

1cd337d

more replacements

b795468

let's try a 5.15 kernel

a10d67c

sharifelgamal commented May 6, 2022

View reviewed changes

pkg/minikube/node/start.go Outdated Show resolved Hide resolved

medyagh requested changes May 6, 2022

View reviewed changes

.github/workflows/update-k8s-versions.yml Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

deploy/iso/minikube-iso/arch/aarch64/package/cri-dockerd-aarch64/cri-dockerd.mk Show resolved Hide resolved

sharifelgamal added 4 commits May 6, 2022 14:52

small fixes

d1829f2

revert unrelated changes

d1d4105

more reverting

b473190

Merge branch 'master' of github.com:kubernetes/minikube into iso-arm64

4294534

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label May 6, 2022

sharifelgamal changed the title ~~WIP: arm64 ISO work~~ Add support for building aarch64 ISO May 6, 2022

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2022

medyagh self-requested a review May 6, 2022 23:01

adding just so so many kernel modules for aarch64

66a8e87

Updating ISO to v1.25.2-1651892774-13762

9c75a9c

medyagh approved these changes May 9, 2022

View reviewed changes

sharifelgamal merged commit acb03c4 into kubernetes:master May 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for building aarch64 ISO #13762

Add support for building aarch64 ISO #13762

sharifelgamal commented Mar 9, 2022 •

edited

Loading

sharifelgamal commented May 6, 2022

minikube-pr-bot commented May 6, 2022

medyagh commented May 7, 2022 •

edited

Loading

sharifelgamal commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-bot commented May 7, 2022

sharifelgamal commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

afbjorklund commented May 7, 2022 •

edited

Loading

sharifelgamal commented May 7, 2022

afbjorklund commented May 7, 2022

k8s-ci-robot commented May 9, 2022

Add support for building aarch64 ISO #13762

Add support for building aarch64 ISO #13762

Conversation

sharifelgamal commented Mar 9, 2022 • edited Loading

sharifelgamal commented May 6, 2022

minikube-pr-bot commented May 6, 2022

medyagh commented May 7, 2022 • edited Loading

sharifelgamal commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-bot commented May 7, 2022

sharifelgamal commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

minikube-pr-bot commented May 7, 2022

afbjorklund commented May 7, 2022 • edited Loading

sharifelgamal commented May 7, 2022

afbjorklund commented May 7, 2022

k8s-ci-robot commented May 9, 2022

sharifelgamal commented Mar 9, 2022 •

edited

Loading

medyagh commented May 7, 2022 •

edited

Loading

afbjorklund commented May 7, 2022 •

edited

Loading