Multiarch - merging manifest fails with httpReadSeeker "failed open - not found"

[ulbi]

I currently try to build a multiarch image and it fails with:

#15 ERROR: httpReadSeeker: failed open: content at https://registry-1.docker.io/v2/myuser/myrep/manifests/sha256:fe138e82236acc7b794fb72bc7311fc651ad4785f6f0a3213a1b8787a14788e0 not found: not found

How can I fix this?

docker buildx version
github.com/docker/buildx v0.10.0 8764628

docker version

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.17.3
 Git commit:        20.10.12-0ubuntu4
 Built:             Mon Mar  7 17:10:06 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.3
  Git commit:       20.10.12-0ubuntu4
  Built:            Mon Mar  7 15:57:50 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu3.1
  GitCommit:
 runc:
  Version:          1.1.0-0ubuntu1.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

Here the complete buildx output:

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 91B done
#2 DONE 0.0s

#3 [linux/amd64 internal] load metadata for docker.io/library/nginx:alpine
#3 ...

#4 [internal] load .dockerignore
#4 transferring context: 2B 0.0s done
#4 DONE 0.1s

#5 [internal] load build definition from Dockerfile
#5 transferring dockerfile: 91B 0.0s done
#5 DONE 0.1s

#3 [linux/amd64 internal] load metadata for docker.io/library/nginx:alpine
#3 DONE 0.3s

#6 [internal] load build context
#6 transferring context: 30B done
#6 DONE 0.0s

#7 [linux/amd64 1/2] FROM docker.io/library/nginx:alpine@sha256:659610aadb34b7967dea7686926fdcf08d588a71c5121edb094ce0e4cdbc45e6
#7 resolve docker.io/library/nginx:alpine@sha256:659610aadb34b7967dea7686926fdcf08d588a71c5121edb094ce0e4cdbc45e6 done
#7 DONE 0.0s

#8 [linux/amd64 2/2] COPY reload.sh /etc/periodic/15min/
#8 CACHED

#9 exporting to image
#9 exporting layers done
#9 exporting manifest sha256:32b166c8b3411ea14e148ce08c0693d303bcfd61701d1f41141ffc36a0e4532a done
#9 exporting config sha256:a84fa96f6db455d4a554332911f001a6942c930a260bc74997f0f1731a2d9c04 done
#9 exporting attestation manifest sha256:9ff7091d741012a228234457b50b7f7b37b013f3271f5b9929e08756fc980232 0.0s done
#9 exporting manifest list sha256:fe138e82236acc7b794fb72bc7311fc651ad4785f6f0a3213a1b8787a14788e0 done
#9 pushing layers
#9 ...

#10 [linux/arm/v7 internal] load metadata for docker.io/library/nginx:alpine
#10 DONE 0.6s

#11 [internal] load build context
#11 transferring context: 30B 0.0s done
#11 DONE 0.1s

#12 [linux/arm/v7 1/2] FROM docker.io/library/nginx:alpine@sha256:659610aadb34b7967dea7686926fdcf08d588a71c5121edb094ce0e4cdbc45e6
#12 resolve docker.io/library/nginx:alpine@sha256:659610aadb34b7967dea7686926fdcf08d588a71c5121edb094ce0e4cdbc45e6 0.1s done
#12 DONE 0.1s

#13 [linux/arm/v7 2/2] COPY reload.sh /etc/periodic/15min/
#13 CACHED

#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:17f8e9b34ca7721bada82d136f172114c311f542520d057a5d1cf4253928bd87 done
#14 exporting config sha256:f307377a68f6583b7ee35a1e1f56a6b430e1d3a1ea955d956e0962678aca81b1 done
#14 exporting attestation manifest sha256:6a1ee2f76c586c9c20aa6dbbf9ec249cea40836b57b777fa372ba9911b647b23
#14 exporting attestation manifest sha256:6a1ee2f76c586c9c20aa6dbbf9ec249cea40836b57b777fa372ba9911b647b23 0.2s done
#14 exporting manifest list sha256:41b8fda0a75459e093fbcac6e67a7e2f6c5161eb853f962b733944b6a159d3b1 0.1s done
#14 ...

#9 exporting to image
#9 pushing layers 1.6s done
#9 pushing manifest for docker.io/myuser/myrep 1.0s done
#9 DONE 2.6s

#14 exporting to image
#14 pushing layers 1.6s done
#14 pushing manifest for docker.io/myuser/myrep 1.0s done
#14 DONE 2.9s

#15 merging manifest list myuser/myrep:sample
#15 ERROR: httpReadSeeker: failed open: content at https://registry-1.docker.io/v2/myuser/myrep/manifests/sha256:fe138e82236acc7b794fb72bc7311fc651ad4785f6f0a3213a1b8787a14788e0 not found: not found
------
 > merging manifest list myuser/myrep:sample:
------

[ulbi]

Just figured out, when using the "QEMU emulation" the build works, but when building with multiple nodes (amd64, armv7, armv6) it fails.

[smira]

Same problem here, disabling provenance with --provenance=false fixes this, only when the image manifest is merged from different multi-arch builders, worked fine with 0.9 buildx/0.10 buildkit:

Logs:

```

#28 exporting to image

1041 | #28 exporting layers
1042 | #28 exporting layers 5.1s done
1043 | #28 ...
1044 |  
1045 | #29 [auth] siderolabs/kubelet:pull,push token for ghcr.io
1046 | #29 DONE 0.0s
1047 |  
1048 | #28 exporting to image
1049 | #28 exporting manifest sha256:b326a138b928c39cc375e80c2d7fba52e00daca8e89d40eceffb16f127ea03e6 done
1050 | #28 exporting config sha256:4959c28513364f05314c03c1b19752c5679eff6477811e177070800c5e2be56c done
1051 | #28 exporting attestation manifest sha256:36d356bce391ce14425835eae34f77bf9d0f10a9b99411e595f2cb4705358f46 done
1052 | #28 exporting manifest list sha256:1ecf2f2b6590dae2fbc7d15da7e021087d9f493ae046ef70264a45912abf6b8c done
1053 | #28 pushing layers
1054 | #28 pushing layers 2.1s done
1055 | #28 pushing manifest for ghcr.io/siderolabs/kubelet
1056 | #28 pushing manifest for ghcr.io/siderolabs/kubelet 2.0s done
1057 | #28 DONE 9.2s
1058 |  
1059 | #30 merging manifest list ghcr.io/siderolabs/kubelet:v1.24.10
1060 | #30 ERROR: httpReadSeeker: failed open: content at https://ghcr.io/v2/siderolabs/kubelet/manifests/sha256:1ecf2f2b6590dae2fbc7d15da7e021087d9f493ae046ef70264a45912abf6b8c not found: not found

</details>

[Romain-Geissler-1A]

Hi,

I don't know if this is related, but it seems that with rather old JFrog's Artifactory Docker registries, when trying to use buildx 0.10.x to push a multi-arch image coming from 2 distinct build nodes, I get errors like:

#48 exporting to image
#48 exporting manifest list sha256:b6020f2bec3d53737d30ee685aef26fa89d9e5f5e3a0307ec49878f2c49e764f 0.1s done
#48 pushing layers
#48 pushing layers 74.0s done
#48 pushing manifest for obeimages-docker-release.nce.dockerhub.rnd.amadeus.net/obe-tools/rhel-base-common
#48 pushing manifest for obeimages-docker-release.nce.dockerhub.rnd.amadeus.net/obe-tools/rhel-base-common 1.2s done
#48 ERROR: failed to push obeimages-docker-release.nce.dockerhub.rnd.amadeus.net/obe-tools/rhel-base-common: failed commit on ref "index-sha256:b6020f2bec3d53737d30ee685aef26fa89d9e5f5e3a0307ec49878f2c49e764f": unexpected status: 400 Bad Request

#26 exporting to image
------
 > exporting to image:
------
ERROR: failed to solve: failed to push obeimages-docker-release.nce.dockerhub.rnd.amadeus.net/obe-tools/rhel-base-common: failed commit on ref "index-sha256:b6020f2bec3d53737d30ee685aef26fa89d9e5f5e3a0307ec49878f2c49e764f": unexpected status: 400 Bad Request
script returned exit code 1

Looking at such error on Artifactory side, it seems it is (at least in some cases) a known issue. I have found for example https://jfrog.atlassian.net/browse/RTFACT-25700 which has been fixed quite some time ago, but which not everyone might use yet (for example we don't).

I tried disabling provenance in my buildx build, and indeed it "fixes" (more works around) my issue.

Just adding my error here as it might help someone else try to investigate new issues with "400 Bad Request" errors which aren't exactly clear on what goes wrong.

Cheers,
Romain