Build multi-arch images using fat manifests

[Prajyot-Parab]

Describe the problem/challenge you have

• Publish velero images for ppc64le Arch

Describe the solution you'd like

• Building multi-arch images using fat manifests

Environment:

• Velero version (use velero version): v1.0.0
• Kubernetes version (use kubectl version): v1.12.4
• Kubernetes installer & version: ICP 3.2.0 EE
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu 16.04

[Prajyot-Parab]

@skriss Your thoughts on how to go ahead with this?

[clnperez]

After talking with @Prajyot-Parab -- it sounds like we'll need to get a better idea of your current build pipeline flow. It looks like you're currently using Travis? That's great b/c it has power nodes available, and we can add pipeline steps with -linux-ppc64le. Prajyot is going to experiment a bit with our internal/enterprise Travis and then we'll work on putting a PR together for this. Any suggestions welcome before we get too far down that path.

[nrb]

The images are currently built on a private Jenkins server, rather than as part of the Travis CI workflow. Only tests are run with Travis.

This needs to be moved, for sure. I don't know that we've settled on Travis yet, as there's some consolidation that needs to happen between VMware and Heptio repos, and there's a risk that Travis queues would be very long in the shared org. Some other open source teams here are looking into how to address this, so I'll talk to them to see what they're doing.

So that may be a dependency before committing to how to build ppc images.

[clnperez]

Aaah, thanks for solving that mystery. We can get you a VM to plug into your jenkins job if you're okay with connecting out to the world. Otherwise, Travis is the easiest to hook a Power build into at the moment. There are also some cloud options if you'd rather not use a pet VM. That just wouldn't be free.

[nrb]

@Prajyot-Parab or @clnperez Do we need a ppc VM to do the build? Is doing a go cross compile and then building the images not enough? I'm not familiar with running on ppc.

[Prajyot-Parab]

@nrb Doing cross-compile will give you binary for PPC, but the building of images needs to be done on a Power machine so that the underlying arch of images is ppc64le. Then we can create manifest from x86 and ppc64le images to get a multi-arch image.
@clnperez Your thoughts on this?

[clnperez]

It looks like in this case a cross-compile would be doable. I took another look at the Dockerfile.ppc64le, and you could change that up to do the wget of the restic binary outside of the container, cross-compile the valero binary for Power, then do a docker cp of both of those straight into the docker image.

[grooverdan]

Or a multistage build where the first part is a native arch with the required downloads etc to cross build the required parts, and the second FROM {image} is a ppc64le/arm/etc arch image where you just COPY --from=0 ... the build artefacts. Without any RUN directives in the second part it won't need qemu or native hardware to build the image.

[Prajyot-Parab]

@nrb @skriss
PR - #1768

[onedr0p]

Any update here?

[autarchprinceps]

We have the same issue with arm64. The image used by the helm chart doesn't have an arm manifest, just amd64. Multiarch images aren't hard to build with buildx.

[zubron]

@autarchprinceps Velero is already using buildx for our multiarch images (this was introduced with #2754) and the 1.5 images were built using this. I've just checked our images on Docker Hub and arm64 images are available for Velero 1.3 and later. The Helm chart uses the images from Docker Hub by default. Can you check what image is being used in your helm chart?