Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multi arch for test-infra/images #16588

Closed
7 tasks
lubinszARM opened this issue Mar 3, 2020 · 36 comments
Closed
7 tasks

Multi arch for test-infra/images #16588

lubinszARM opened this issue Mar 3, 2020 · 36 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/testing Categorizes an issue or PR as relevant to SIG Testing.

Comments

@lubinszARM
Copy link
Contributor

lubinszARM commented Mar 3, 2020 •
edited
Loading

This issue is for pushing multi-architecture images for test-infra/images, so that the specific jobs can be scheduled by Prow into Arm64 nodes.

  • kubekins-test
  • builder, google/cloud-sdk

If a typical k8s-e2e-kind-test prow job was scheduled into an Arm64 k8s cluster, a POD contains at least the following images, so we should make them to support Arm64:

  • kubekins-e2e
  • bootstrap
  • sidecar
  • initupload
  • clonerefs
@lubinszARM lubinszARM added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 3, 2020
@lubinszARM
Copy link
Contributor Author

cc @BenTheElder @zhlhahaha

@BenTheElder
Copy link
Member

This seems quite a bit ahead of ourselves without any plan for Prow on such nodes.

cc @dims @mkumatag

@mkumatag
Copy link
Member

mkumatag commented Mar 3, 2020

Interesting, I was thinking about this for quite some time. Can we have individual architecture's cluster running anywhere in the word and prow scheduling respective arch's job into those clusters and get things done?

@BenTheElder
Copy link
Member

BenTheElder commented Mar 4, 2020 via email

@lubinszARM
Copy link
Contributor Author

This issue is used to track it.
We will test it in our local hybrid CI cluster firstly.
Finally we will update this issue and then decide whether to do this.
Thanks.

@zhlhahaha
Copy link
Contributor

zhlhahaha commented Mar 20, 2020 •
edited
Loading

Hi, @BenTheElder @mkumatag @dims , here is some update, issues and plans about prow test on ARM. Please feel free to tell me if there is anything not clear. Looking forward your suggestion.

Update:

  • We deployed a prow cluster on x86 and a kubernetes cluster on ARM.
  • enable prow to assign test jobs to ARM k8s cluster
  • Set a presubmit jobs, "pull-kubernetes-conformance-image-test", on ARM k8s cluster, it can be run successfully and all kind-conformance test passed, result has upload to gs

Issues:

  • The kubekins-e2e image for arm platform is build by ourselves.
  • In the prow jobs, we still use bootstrap.py because we are not able to use podutils as the images, bootstrap, sidecar, initupload, clonerefs only support x86 platform.
  • In order to trigger the presubmit jobs, I use my private kubernetes library in github.com.
  • The script "kind-conformance-image-e2e.sh" is only support x86 platform. I submit a patch for mutliarch support, but it is not enough. As the version of kind in the script is 0.6.0, and the binary do not works well on Arm Platform. We use kind that compiled from most resent source code.

Plans:

  • enable kubekins-e2e image build for multi-arch
  • enable bootstrap, sidecar, initupload, clonerefs build for multi-arch
  • looking forward new release of kind

@BenTheElder
Copy link
Member

I still don't think we need a prow on ARM at all, we can let a prow job on the normal build cluster SSH / whatever to ARM infra the same as we do when we bring up cluster on AWS / GCP.

@lubinsz
Copy link

lubinsz commented Mar 24, 2020 •
edited
Loading

Hi @BenTheElder
According to our research, prow cluster is still on x86, and we can create an arm64 build cluster, and let some typical test prow jobs running on Arm64, such as pull-kubernetes-conformance-image-test.
So, in my opinion, we should make some typical prow-jobs related images to support Arm64.

The following figure is Prow's microservice architecture for the test job of 'pull-kubernetes-conformance-image-test' on Arm64.
image

@zhlhahaha zhlhahaha mentioned this issue May 28, 2020
Merged
@lubinszARM
Copy link
Contributor Author

cc @Jingzhao123

@Jingzhao123 Jingzhao123 mentioned this issue Jun 17, 2020
Closed
@lubinszARM lubinszARM mentioned this issue Jun 29, 2020
Merged
@lubinsz lubinsz mentioned this issue Jul 7, 2020
Closed
@zhlhahaha
Copy link
Contributor

Hi @BenTheElder @spiffxp @mkumatag
Can we enable prow utility images for multi-arch?
Here are some reasons why we need prow utility images running on multi-arch.

  1. The k8s conformance test running on ARM which you can see in https://testgrid.k8s.io/sig-node-arm64, are triggered by shell script currently. And we want to sync the test flow with kuternetes test-infra by using prow.
  2. Some other projects like kubevirt, kubeflow, they all use prow for their CI lane. We are trying to enable ARM CI testing for these projects.

@rmohr
Copy link
Contributor

rmohr commented Jul 30, 2020 •
edited
Loading

I still don't think we need a prow on ARM at all, we can let a prow job on the normal build cluster SSH / whatever to ARM infra the same as we do when we bring up cluster on AWS / GCP.

This can definitely work. However consider how much simpler it is to onboard different architectures for opensource projects, if you just pass a new cluster context to prow and put a label on your job.

As far as I can see, it would mostly just be necessary to cross-compile a few plain docker binaries. clonerefs may need a multiarch base image with git (building that with qemu-user-static should be streight forward. We do that in kubevirt too on our amd64 machines). The bootstrap image is in my opinion not really needed. It would be about

sidecar
initupload
clonerefs

only.

@BenTheElder would you see a way where we could maintain this in this repo? We basically have the same issue with ppc64 too btw.

@zhlhahaha zhlhahaha mentioned this issue Sep 2, 2020
Closed
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 28, 2020
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 27, 2020
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@k8s-ci-robot
Copy link
Contributor

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@LorbusChris
Copy link
Contributor

LorbusChris commented May 27, 2021 •
edited
Loading

Looks like this hasn't been solved yet
/reopen

For OpenShift, we're currently in the process of standing up arm64 test infra, and I think at least the following images are going to be needed:

  • clonerefs
  • entrypoint
  • initupload
  • sidecar

@BenTheElder is this still considered something that could be done and maintained in this repo?

cc @stevekuznetsov

Edit: @stevekuznetsov, could you reopen this issue please

@k8s-ci-robot
Copy link
Contributor

@LorbusChris: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

Looks like this hasn't been solved yet
/reopen

For OpenShift, we're currently in the process of standing up arm64 test infra, and I think at least the following images are going to be needed:

  • clonerefs
  • entrypoint
  • initupload
  • sidecar

@BenTheElder is this still considered something that could be done and maintained in this repo?

cc @stevekuznetsov

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@BenTheElder
Copy link
Member

I think you're going to find it a bit challenging to produce these in a multi-arch way with the current build tools versus at least one arch at a time and some post processing. But ultimately it's the call of the active prow/OWNERS as to what's acceptable.

The original context was ensuring kubernetes CI has arm coverage for which we didn't need this at all (versus creating clusters under test on ARM), that conversation is what brought me here, I have no vested interest one way or another otherwise.

@BenTheElder BenTheElder reopened this May 27, 2021
@BenTheElder BenTheElder removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label May 27, 2021
This was referenced Aug 18, 2021
Merged
Merged
This was referenced Aug 31, 2021
Merged
Merged
Merged
@howardjohn howardjohn mentioned this issue Sep 15, 2021
Closed
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 8, 2021
@chendave
Copy link
Member

chendave commented Nov 8, 2021

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 8, 2021
@chendave chendave mentioned this issue Jan 14, 2022
Closed
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 6, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 8, 2022
@chendave
Copy link
Member

/remove-lifecycle rotten
/remove-lifecycle frozen

someone has interests on this can still input their comments here.

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Mar 31, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jun 29, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jul 29, 2022
@chendave
Copy link
Member

/remove-lifecycle stale
/lifecycle frozen

@k8s-ci-robot k8s-ci-robot added lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. and removed lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. labels Jul 29, 2022
@raelga
Copy link
Member

raelga commented Sep 25, 2024

Is there a any updates on ARM64 for Prow?

@BenTheElder
Copy link
Member

Prow can run test pods on arm64 and doesn't live in this repo anymore.

@BenTheElder
Copy link
Member

https://github.com/kubernetes-sigs/prow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Categorizes issue or PR as related to a new feature. lifecycle/frozen Indicates that an issue or PR should not be auto-closed due to staleness. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests