Building multiarch docker image: amd, s390x #251

namrata-ibm · 2021-08-30T13:51:16Z

This PR aims at building and pushing multi-arch image on docker hub.
New architectures can be easily added to the platforms to generate manifests for others.

namrata-ibm · 2021-08-30T13:51:52Z

@jhump could you please check and let me know your comments? Thank you!

namrata-ibm · 2021-09-15T11:22:29Z

@jhump Could you please have a look?

jhump · 2021-09-15T21:42:11Z

I like the idea. This could also be used to resolve #234.

However, I have no access to hardware for these platforms, so I do not know how I could test the images. That makes me nervous in supporting them.

I know the same argument could be made of the compiled binaries in the GitHub release artifacts, which I have included (even for platforms on which I can't test). I guess I trust Go cross-compilation results, having used them a lot. However, I have never used multi-arch support in Docker, and it's even marked as an "experimental" feature. So I guess I'm not as trusting.

Do you have a proposal for how I could verify these images?

namrata-ibm · 2021-09-16T10:57:59Z

@jhump Thank you for your feedback.

We do understand your concern, however, many repositories have enabled buildx for generating multi-arch images now(some of them enabled by our team) - kubernetes , jaeger , kubernetes-csi amongst the few I can quickly list here.

Regarding the hardware access to test images, we can provide s390x VMs to you to validate the images at your end.
Please let me know if it helps.

jhump · 2021-09-20T15:58:16Z

releasing/do-release.sh

-rm VERSION
+
+# Enable qemu and binfmt support
+$PREFIX docker run --privileged --rm tonistiigi/binfmt:qemu-v6.1.0 --install all


What is this for? Does this install something on the host that is needed for the buildx stuff below? If so, maybe elaborate in the comment above.

Updated the comment, Also had missed earlier to mention about buildx added in docker 19.03 onward. Updated the same.

jhump · 2021-09-20T15:59:58Z

releasing/do-release.sh

-$PREFIX docker push "fullstorydev/grpcurl:${VERSION}"
-$PREFIX docker tag "fullstorydev/grpcurl:${VERSION}" fullstorydev/grpcurl:latest
-$PREFIX docker push fullstorydev/grpcurl:latest
+$PREFIX docker buildx build --platform linux/amd64,linux/s390x --tag fullstorydev/grpcurl:${VERSION} --tag fullstorydev/grpcurl:latest --push --progress plain --no-cache .


Let's add linux/arm64, too, to resolve #234.

BTW, I don't have any experience using multi-platform docker images. Since this statement uses tags like latest and ${VERSION}, I guess the platform does not need to be part of the tag, and that Docker hub (or other image registries) has a separate way of partitioning images for different platforms?

Let's add linux/arm64, too, to resolve #234.

I have added arm64, it should work, however please note that I do not have access to verify image on arm. Someone using arm can help us validate. I have verified the images on amd and s390x so far.
Also please note that 'go build' step takes higher time for arm and s390x than amd as of now. Hopefully it will improve in future releases.

BTW, I don't have any experience using multi-platform docker images. Since this statement uses tags like latest and ${VERSION}, I guess the platform does not need to be part of the tag, and that Docker hub (or other image registries) has a separate way of partitioning images for different platforms?

That's right, docker uses manifest lists for a set of platforms and pulls the correct image based on where it is running. This describes it well.
I have kept the same image names and tags (latest and $VERSION), this will ensure existing amd users are not affected!

I will see if @pquerna can test an ARM image when I build and push one (that's who requested ARM support in docker).

namrata-ibm · 2021-09-23T12:17:16Z

Thanks @jhump for your support!

jhump · 2021-10-07T19:54:35Z

@namrata-ibm, I'm having problems with this stuff.

I'm trying to make a release. The first run failed (a flaky error during the go build step for s390x, due to a TLS handshake issue talking to proxy.golang.org). When I tried to re-run it, I get this error:

installing: mips64 qemu-mips64 already registered
installing: mips64le qemu-mips64el already registered
{
  "supported": [
    "linux/amd64",
    "linux/arm64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "qemu-aarch64",
    "qemu-arm",
    "qemu-mips64",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x"
  ]
}
existing instance for multiarch-builder but no append mode, specify --node to make changes for existing instances

I don't know any of this toolchain (which is another reason I was initially hesitant to accept a patch like this).

Can you please advise ASAP?

jhump · 2021-10-07T20:12:07Z

I think this fixes my problem: #264

I've played around with, using docker buildx ls, docker buildx rm, etc, making sure that PR seems to be the solution: it creates the context if it does not exist and is basically a no-op (AFAICT) if it does already exist.

If you know of a better solution, please let me know.

jhump · 2021-10-07T20:34:56Z

Ugh, another botched release. It looks like I simply cannot build this for s390x. Apparently it's an issue in qemu on OS X. The work-around I've seen is to use -cpu qemu,vx=off, however I cannot find any documentation in docker or binfmt as to how I might supply that command-line arg to qemu during s390x builds.

So I'm afraid I am reverting part of this pull request, removing support for s390x for the docker images that get pushed. The next release will still have published binaries for s390x. But if you want official docker images, I'll need more help in a workable solution.

namrata-ibm · 2021-10-08T10:41:27Z

@jhump apologies for the late response.. time zone difference!

I think this fixes my problem: #264

I've played around with, using docker buildx ls, docker buildx rm, etc, making sure that PR seems to be the solution: it creates the context if it does not exist and is basically a no-op (AFAICT) if it does already exist.

If you know of a better solution, please let me know.

Above looks good.

About build for s390x, I tried to run the do-release.sh v1.8.5 from 1.8.5 tag as well as master and it passed for me, also multiarch image was pushed successfully:

#21 [linux/s390x builder 6/6] RUN go build -o /grpcurl     -ldflags "-w -ext...
#21 ...

#30 [linux/arm64 builder 6/6] RUN go build -o /grpcurl     -ldflags "-w -ext...
#30 DONE 590.4s

#21 [linux/s390x builder 6/6] RUN go build -o /grpcurl     -ldflags "-w -ext...
#21 ...

#31 [linux/arm64 stage-1 1/4] COPY --from=builder /etc/ssl/certs/ca-certific...
#31 CACHED

#32 [linux/arm64 stage-1 2/4] COPY --from=builder /etc/passwd /etc/passwd
#32 CACHED

#33 [linux/arm64 stage-1 3/4] COPY --from=builder /grpcurl /bin/grpcurl
#33 DONE 0.2s

#21 [linux/s390x builder 6/6] RUN go build -o /grpcurl     -ldflags "-w -ext...
#21 DONE 740.6s

#22 [linux/s390x stage-1 1/4] COPY --from=builder /etc/ssl/certs/ca-certific...
#22 CACHED

#23 [linux/s390x stage-1 2/4] COPY --from=builder /etc/passwd /etc/passwd
#23 CACHED

#24 [linux/s390x stage-1 3/4] COPY --from=builder /grpcurl /bin/grpcurl
#24 DONE 0.2s

#34 exporting to image
#34 exporting layers
#34 exporting layers 2.1s done
#34 exporting manifest sha256:692fb5ea58520ccd4d20f1b9875571ad2d2f183dae0303844b22c770354fdb08 0.0s done
#34 exporting config sha256:a7fc0055294eb937600593a4a50245aa3f381419daf34b2be84f90ac34f59ccd 0.0s done
#34 exporting manifest sha256:aaf8813e2126386c3bf3dc194648d086f10b3404b3234fdd0f497de49140c532 0.0s done
#34 exporting config sha256:012fb0775dea8aee1122f31a8f812f75dbb429b2ca1b18ea41407d2af796ee38 0.0s done
#34 exporting manifest sha256:7b170f7348ece41f5d226d86d5be0f118edc0da0cc5ca3ef210b83225eaaa805 0.0s done
#34 exporting config sha256:eb75db15b5ad7991a4c2dc53fd9c4b4fe987d54d3f2d853781fb144d50f1c3ab
#34 exporting config sha256:eb75db15b5ad7991a4c2dc53fd9c4b4fe987d54d3f2d853781fb144d50f1c3ab 0.0s done
#34 exporting manifest list sha256:b6126bd59848434e2b08fb45b00c298af8d5eb1b5ff669b82cbefbff9a1eed10 0.0s done
#34 pushing layers
#34 pushing layers 3.0s done
#34 pushing manifest for docker.io/xx/grpcurl:v1.8.5@sha256:b6126bd59848434e2b08fb45b00c298af8d5eb1b5ff669b82cbefbff9a1eed1
#34 pushing manifest for docker.io/xx/grpcurl:v1.8.5@sha256:b6126bd59848434e2b08fb45b00c298af8d5eb1b5ff669b82cbefbff9a1eed11.4s done
#34 pushing layers 0.8s done
#34 pushing manifest for docker.io/xx/grpcurl:latest@sha256:b6126bd59848434e2b08fb45b00c298af8d5eb1b5ff669b82cbefbff9a1eed1
#34 pushing manifest for docker.io/xx/grpcurl:latest@sha256:b6126bd59848434e2b08fb45b00c298af8d5eb1b5ff669b82cbefbff9a1eed10.7s done
#34 DONE 8.2s

Could you please give some details on the error faced?
I have tried it on Ubuntu 18.04 x86_64 machine by running only the # Docker release section from do-release.sh and commented the # GitHub release and # Homebrew release, although this shouldn't cause any issues.

jhump · 2021-10-08T13:15:35Z

I have tried it on Ubuntu 18.04 x86_64

In my last message, I pointed out that it fails on OS X.

Sorry I that I forgot to put a link to the actual issue I'm running into in my previous comment. Here it is:
golang/go#40949

I thought the error was flaky, like a TLS issue with the proxy.golang.org server. But it's actually a bug in qemu for the s390x platform on OS X. So I cannot create s390x builds because building the image fails during the go build ... step.

namrata-ibm · 2021-10-12T11:14:27Z

@jhump Thanks, the golang issue which you have added, was also observed on Intel. However it has been fixed in latest qemu and this PR uses the same via this. So not sure why this is appearing at your end. Will check if this is OS related.

namrata-ibm · 2021-10-13T05:47:55Z

@jhump Could you please share details about your system(model/year) to see if I can reproduce it?

jhump · 2021-10-14T16:55:19Z

I'm using a mid-2015 Macbook Pro, 15", with OS X 10.14.6 (Mojave).

The bug report I posted above suggests it is related to OS X. It may only be Intel-chip Macbooks (so maybe this not an issue for newer ones that use M1 chips?).

Building multiarch docker image: amd, s390x

5e3b216

jhump reviewed Sep 20, 2021

View reviewed changes

Addressing review comments

a9a2b5d

jhump approved these changes Sep 23, 2021

View reviewed changes

jhump merged commit 2e9ba50 into fullstorydev:master Sep 23, 2021

jhump mentioned this pull request Oct 7, 2021

can't build s390x docker images; skip for now #265

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building multiarch docker image: amd, s390x #251

Building multiarch docker image: amd, s390x #251

namrata-ibm commented Aug 30, 2021

namrata-ibm commented Aug 30, 2021

namrata-ibm commented Sep 15, 2021

jhump commented Sep 15, 2021 •

edited

Loading

namrata-ibm commented Sep 16, 2021

jhump Sep 20, 2021

namrata-ibm Sep 23, 2021

jhump Sep 20, 2021

namrata-ibm Sep 23, 2021

jhump Sep 23, 2021 •

edited

Loading

namrata-ibm commented Sep 23, 2021

jhump commented Oct 7, 2021 •

edited

Loading

jhump commented Oct 7, 2021

jhump commented Oct 7, 2021

namrata-ibm commented Oct 8, 2021

jhump commented Oct 8, 2021

namrata-ibm commented Oct 12, 2021

namrata-ibm commented Oct 13, 2021

jhump commented Oct 14, 2021

Building multiarch docker image: amd, s390x #251

Building multiarch docker image: amd, s390x #251

Conversation

namrata-ibm commented Aug 30, 2021

namrata-ibm commented Aug 30, 2021

namrata-ibm commented Sep 15, 2021

jhump commented Sep 15, 2021 • edited Loading

namrata-ibm commented Sep 16, 2021

jhump Sep 20, 2021

Choose a reason for hiding this comment

namrata-ibm Sep 23, 2021

Choose a reason for hiding this comment

jhump Sep 20, 2021

Choose a reason for hiding this comment

namrata-ibm Sep 23, 2021

Choose a reason for hiding this comment

jhump Sep 23, 2021 • edited Loading

Choose a reason for hiding this comment

namrata-ibm commented Sep 23, 2021

jhump commented Oct 7, 2021 • edited Loading

jhump commented Oct 7, 2021

jhump commented Oct 7, 2021

namrata-ibm commented Oct 8, 2021

jhump commented Oct 8, 2021

namrata-ibm commented Oct 12, 2021

namrata-ibm commented Oct 13, 2021

jhump commented Oct 14, 2021

jhump commented Sep 15, 2021 •

edited

Loading

jhump Sep 23, 2021 •

edited

Loading

jhump commented Oct 7, 2021 •

edited

Loading