How to replace docker images of kube-scheduler(apiserver etc.) when creating cluster

[Illyrix]

I'm working on an ARM64 machine and has built node-image for ARM64 successfully.
I've got error while executing kind create cluster --image kindest/node:latest --retain

There were some unexpected exited containers (after failed to create cluster), and the reason was possibly these images were built for amd64 but run on arm64.

Is there a way to replace these images of containers(kube-controller-manager, kube-apiserver, kube-scheduler, etcd, pause) by ARM64 version (such as kube-apiserver-arm64)?

[BenTheElder]

Are you sure you built for arm? There should not be the architecture in the image name.

…

On Thu, Feb 20, 2020, 00:52 Illyrix ***@***.***> wrote: I'm working on an ARM64 machine and has built node-image for ARM64 successfully. I've got error while executing kind create cluster --image kindest/node:latest --retain [image: ***@***.*** <https://user-images.githubusercontent.com/12008675/74916515-f5df3280-5400-11ea-8817-a55eddefea65.png> There were some unexpected exited containers (after failed to create cluster), and the reason was possibly these images were built for amd64 but run on arm64. [image: ***@***.*** <https://user-images.githubusercontent.com/12008675/74914363-48b6eb00-53fd-11ea-9304-4d8be539eed0.png> [image: ***@***.*** <https://user-images.githubusercontent.com/12008675/74914532-8ddb1d00-53fd-11ea-90b2-53cd882d1575.png> Is there a way to replace these images of containers(kube-controller-manager, kube-apiserver, kube-scheduler, etcd, pause) by ARM64 version (such as kube-apiserver-arm64)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1344?email_source=notifications&email_token=AAHADK3GBIJZ4L3ULRZEEZLRDZADJA5CNFSM4KYKBZA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IO5F2HQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK5VQA4PLL3LTXY6W5TRDZADJANCNFSM4KYKBZAQ> .

[Illyrix]

Are you sure you built for arm? There should not be the architecture in the image name.
…
On Thu, Feb 20, 2020, 00:52 Illyrix @.> wrote: I'm working on an ARM64 machine and has built node-image for ARM64 successfully. I've got error while executing kind create cluster --image kindest/node:latest --retain [image: @. https://user-images.githubusercontent.com/12008675/74916515-f5df3280-5400-11ea-8817-a55eddefea65.png There were some unexpected exited containers (after failed to create cluster), and the reason was possibly these images were built for amd64 but run on arm64. [image: @.*** https://user-images.githubusercontent.com/12008675/74914363-48b6eb00-53fd-11ea-9304-4d8be539eed0.png [image: @.*** https://user-images.githubusercontent.com/12008675/74914532-8ddb1d00-53fd-11ea-90b2-53cd882d1575.png Is there a way to replace these images of containers(kube-controller-manager, kube-apiserver, kube-scheduler, etcd, pause) by ARM64 version (such as kube-apiserver-arm64)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1344?email_source=notifications&email_token=AAHADK3GBIJZ4L3ULRZEEZLRDZADJA5CNFSM4KYKBZA2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IO5F2HQ>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHADK5VQA4PLL3LTXY6W5TRDZADJANCNFSM4KYKBZAQ .

If i re-tag "pause" docker tag gcr.io/google_containers/pause-arm64:3.2 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1, "pause" works well

But I think it is not a good solution. "kube-scheduler"(and others) are distinguished by hash-tag not readable version tag, so it's hard to be solved by this trick way

[zhouqiang-cl]

@BenTheElder May I ask can kind support arm64 now? seems it is still in progress

[aojea]

IIRC Kind is building for ARM ... another thing is that we don't have CI so it can not be supported because there is no way to test it
#188

Can you try building kind and the node images locally in an ARM architecture?

[BenTheElder]

arm is nominally supported BUT you must build your own images, we are not trying to do a manifest image of the node or base images currently and we don't have upstream CI.

there's been pretty minimal demand, and non-trivial expense to running this.

you need to:

• build a base image
• use this base image to build a node image
• use this node image when running create cluster

[BenTheElder]

we do build the kind binary for multiple platforms but pre-built node & base image are amd64. the haproxy and kindnetd images are multi-arch.

[BenTheElder]

If i re-tag "pause" docker tag gcr.io/google_containers/pause-arm64:3.2 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1, "pause" works well
the pause:3.1 image is expected to be a manifest list, and on the current platform it should match.

how are you building this image? I can't tell from the log screenshots.
and does that pause:3.1 match the upstream registry?

But I think it is not a good solution. "kube-scheduler"(and others) are distinguished by hash-tag not readable version tag, so it's hard to be solved by this trick way

huh? we definitely load / reference them using a tag and so does kubeadm ...

[BenTheElder]

kube-scheduler etc. should be from the locally built images during kind build node-image, which invokes the upstream kubernetes build process. they should be for the current architecture.

only a few images like etcd are pulled instead of built, and those are multi-arch pre-built upstream images.

[zhouqiang-cl]

arm is nominally supported BUT you must build your own images, we are not trying to do a manifest image of the node or base images currently and we don't have upstream CI.

there's been pretty minimal demand, and non-trivial expense to running this.

you need to:

• build a base image
• use this base image to build a node image
• use this node image when running create cluster

thank you @BenTheElder . we will try to do it

[BenTheElder]

#1346