buildx create crashes from a 401 Unauthorized

[pjonsson]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Running docker -l debug --config .ro_config buildx create --name ci-builder --driver docker-container --driver-opt image=moby/buildkit:latest --bootstrap crashes with a 401 Unauthorized.

Expected behaviour

I would expect the command to create a builder called ci-builder using the moby/buildkit:latest image. If this command requires access to other images, I would expect the docker login on the line before to give that access.

Actual behaviour

I don't know, but I provide logs containing debug output in the box below.

Buildx version

github.com/docker/buildx v0.10.4 c513d34

Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 10
  Running: 1
  Paused: 0
  Stopped: 9
 Images: 121
 Server Version: 23.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-71-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 62.8GiB
 Name: gitlab-runner-1
 ID: REDACTED
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
  https://REDACTED/
 Live Restore Enabled: false

Builders list

$ docker --config .ro_config buildx ls (on the CI-runner after the job failed)
NAME/NODE     DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
ci-builder    docker-container                             
  ci-builder0 unix:///var/run/docker.sock running v0.11.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, 
linux/386
default *     docker                                       
  default     default                     running 23.0.5   linux/amd64, linux/amd64/v2, linux/amd64/v3, 
linux/386

Configuration

Empty config, the .ro_config directory from the previous run is removed by the CI before the run starts.

Build logs

$ echo $RO_PASSWORD | docker --config .ro_config login $CI_REGISTRY -u $RO_USER --password-stdin
WARNING! Your password will be stored unencrypted in .ro_config/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker -l debug --config .ro_config buildx create --name ci-builder --driver docker-container --driver-opt image=moby/buildkit:latest --bootstrap
#1 [internal] booting buildkit
#1 starting container buildx_buildkit_ci-builder0
#1 20.95 time="2023-04-18T15:09:19Z" level=warning msg="using host network as the defaultime="2023-04-18T15:09:19Z" level=warning msg="using host network as the default"
#1 20.95 time="2023-04-18T15:09:20Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#1 20.95 �\0\0\0\0\0\0dtime="2023-04-18T15:09:20Z" level=info msg="found 1 workers, default=\"<key-like sequence>\""
#1 20.95 �\0\0\0\0\0\0`time="2023-04-18T15:09:20Z" level=warning msg="currently, only the default worker can be used."
#1 20.95 �\0\0\0\0\0\0\time="2023-04-18T15:09:21Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
#1 20.95 �\0\0\0\0\0\0ytime="2023-04-18T15:09:44time="2023-04-18T15:09:20Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#1 20.95 time="2023-04-18T15:09:20Z" level=warning msg="currently, only the default worker can be used."
#1 20.95 �\0\0\0time="2023-04-18T15:09:20Z" level=warning msg="currently, only the default worker can be used."
#1 20.95 time="2023-04-18T15:09:44Z" level=info msg="trying next host - response was http.StatusNotFotime="2023-04-18T15:09:44Z" level=info msg="trying next host - response was http.StatusNotFound" host=$CI_REGISTRY
#1 20.95 time="2023-04-18T15:15:16Z" level=info msg="trying next host - response was http.StatusNotFound" host=$CI_REGISTRY
#1 20.95 time="2023-04-18T15:15:16Z" level=info msg="trying next host - response was http.StatusNotFound" host=$CI_REGISTRY
#1 20.95 time="2023-04-18T16:40:51Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 20.95 time="2023-04-18T16:40:51Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 20.95 time="2023-04-19T00:28:48Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 20.95 time="2023-04-19T00:28:48Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 20.95 time="2023-04-19T01:03:26Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 20.95 time="2023-04-19T01:03:26Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T15:49:26Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 starting container buildx_buildkit_ci-builder0 21.0s done
#1 21.04 time="2023-04-19T15:49:26Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T17:13:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T17:13:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T17:13:40Z" level=error msg="fatal error: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/special-python:3.8-slim: pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T17:13:40Z" level=error msg="fatal error: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/special-python:3.8-slim: pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 
#1 21.04 �\0\0\0\0\0\0�goroutine 1 [running]:
#1 21.04 �\0\0\0\0\0\0main.main()
#1 21.04 �\0\0\0\0\0\0D	/src/frontend/dockerfile/cmd/dockerfile-frontend/main.go:31 +0x27e
#1 21.04 �\0\0\0\0\0\0�panic: $CI_REGISTRY (except last character)
#1 21.04 main.main()
#1 21.04 �\0\0\0\0\0\0D	/smain.main()
#1 21.04 panic: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/special-python:3.8-slim: pulling panic: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/special-python:3.8-slim: pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized
#1 21.04 ggoroutine 1 [running]:
#1 21.04 	/src/fronte	/src/frontend/dockerfile/cmd/dockerfile-frontend/main.go:31 +0x27e
#1 21.04 time="2023-04-19T17:13:41Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T17:13:41Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T17:13:46Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T17:13:46Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T17:58:30Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T17:58:30Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T18:29:39Z" level=info msg="trying next host" error="failed to do request: context cancetime="2023-04-19T18:29:39Z" level=info msg="trying next host" error="failed to do request: context canceled" host=$CI_REGISTRY spanID=6fa0730179fde137 traceID=d0013748e0daea4bfaf996aef8eb87a1
#1 21.04 panic: $CI_REGISTRY/$CI_BASE_PATH/special-python:3.8-slim: pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized
�\0\0\0\0\0\0�
�\0\0\0\0\0\0�goroutine 1 [running]:
�\0\0\0\0\0\0mainpanic: $CI_REGISTRY/$CI_BASE_PATH/special-python:3.8-slim: pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized
#1 21.04 ggoroutine 1 [running]:
#1 21.04 	/src/fronte	/src/frontend/dockerfile/cmd/dockerfile-frontend/main.go:31 +0x27e
#1 21.04 time="2023-04-19T18:29:43Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 �\0\0\0\0\0\0htimtime="2023-04-19T18:29:43Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests 3.8-slim]: 401 Unauthorized"
#1 21.04 time="2023-04-19T19:25:31Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T19:25:31Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.04 time="2023-04-19T19:32:15Z" level=info msg="stopping server"
#1 21.04 �\0\0\0\0\0\0�buildkitd: context canceled
#1 21.04 \0\0\0\0\0\0\0time="2023-04-19T19:32:15Z" level=info msg="stopping server"
#1 21.04 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0time="2023-04-19T20:57:37Z" level=warning msg="using host network as the defaultime="2023-04-19T20:57:37Z" level=warning msg="using host network as the default"
#1 21.04 time="2023-04-19T20:58:21Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#1 21.04 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0time="2023-04-19T20:58:21Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
#1 21.04 time="2023-04-19T20:58:21Z" level=warning msg="currently, only the default worker can be used."
#1 21.04 �\0\0\0time="2023-04-19T20:58:21Z" level=warning msg="currently, only the default worker can be used."
#1 21.05 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0time="2023-04-20T01:18:55Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T01:18:55Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T08:28:12Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T08:28:12Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T08:29:00Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T08:29:00Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T09:22:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T09:22:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T10:53:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T10:53:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T10:54:44Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T10:54:44Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T12:01:40Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T12:01:40Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T19:45:59Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T19:45:59Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T19:47:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T19:47:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T20:36:35Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-20T20:36:35Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T15:59:13Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T15:59:13Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T16:00:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T16:00:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T23:24:24Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-21T23:24:24Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T09:30:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T09:30:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T09:31:40Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T09:31:40Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T10:42:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T10:42:09Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T20:15:23Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T20:15:23Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T20:16:31Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T20:16:31Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T23:16:19Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T23:16:19Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T23:23:38Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0�time="2023-04-22T23:23:47Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Untime="2023-04-22T23:23:38Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T23:36:01Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-22T23:47:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0�ttime="2023-04-22T23:36:01Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-22T23:47:35Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: time="2023-04-22T23:47:35Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: not found"
#1 21.05 time="2023-04-23T00:27:46Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T00:27:46Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T10:31:07Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-23T10:32:06Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0httime="2023-04-23T10:31:07Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T10:32:11Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T10:32:11Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T23:20:51Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-23T23:20:51Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:04:30Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:04:30Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:11:34Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: time="2023-04-24T23:11:34Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: not found"
#1 21.05 time="2023-04-24T23:19:11Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: time="2023-04-24T23:19:11Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: not found"
#1 21.05 time="2023-04-24T23:35:07Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:35:07Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:43:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-24T23:48:28Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0�ttime="2023-04-24T23:43:18Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-24T23:48:31Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: time="2023-04-24T23:48:31Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: not found"
#1 21.05 time="2023-04-25T10:32:01Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-25T10:32:01Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-25T12:10:25Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-25T12:10:25Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-25T12:12:05Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-25T12:12:05Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T00:03:06Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T00:03:06Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T09:30:34Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T09:30:34Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T09:31:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T09:31:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T11:14:01Z" level=error msg="fatal error: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/$OTHERIMAGE: pulling from host $CI_REGISTRY failed with status code [manifests $REV_PREFIXtime="2023-04-26T11:14:01Z" level=error msg="fatal error: $CI_REGISTRY/$CI_REGISTRY_BASEPATH/$OTHERIMAGE:$REV: pulling from host $CI_REGISTRY failed with status code [manifests $REV]: 401 Unauthorized"
#1 21.05 
#1 21.05 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0
#1 21.05 main.main()
#1 21.05 �\0\0\0\0\0\0D	/smain.main()
#1 21.05 time="2023-04-26T11:14:02Z" level=error msg="/moby.buildkit.v1.fronttime="2023-04-26T11:14:02Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = pulling from host $CI_REGISTRY failed with status code [manifests $REV]: 401 Unauthorized"
#1 21.05 time="2023-04-26T12:27:16Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-26T12:28:36Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04time="2023-04-26T12:27:16Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T12:28:50Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T12:28:50Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T20:18:48Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="202time="2023-04-26T20:18:48Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T20:20:19Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T20:20:19Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T23:13:23Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T23:13:23Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T23:20:49Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0�time="2023-04-26T23:21:27Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Untime="2023-04-26T23:20:49Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T23:32:33Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-26T23:44:59Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0�ttime="2023-04-26T23:32:33Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-26T23:45:01Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: time="2023-04-26T23:45:01Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = content digest sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6: not found"
#1 21.05 time="2023-04-27T00:26:52Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T00:26:52Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T07:37:57Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0htime="2023-04-27T07:39:25Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 �\0\0\0\0\0\0httime="2023-04-27T07:37:57Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T07:39:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T07:39:32Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T11:53:21Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T11:53:21Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:37:41Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:37:41Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:38:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:38:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 buildkitd: context canceled
#1 21.05 �\0\0\0\0\0\0Otime="2023-04-27T17:17:53buildkitd: context canceled
#1 21.05 time="2023-04-27T17:17:53Z" level=warning msg="using host network as the defaultime="2023-04-27T17:17:53Z" level=warning msg="using host network as the default"
#1 21.05 error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory"
#1 ERROR: exit code 1
------
 > [internal] booting buildkit:
#1 21.05 time="2023-04-27T11:53:21Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T11:53:21Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:37:41Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:37:41Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:38:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 time="2023-04-27T12:38:45Z" level=warning msg="forcibly turning on oci-mediatype mode for attestations"
#1 21.05 buildkitd: context canceled
#1 21.05 �\0\0\0\0\0\0Otime="2023-04-27T17:17:53buildkitd: context canceled
#1 21.05 time="2023-04-27T17:17:53Z" level=warning msg="using host network as the defaultime="2023-04-27T17:17:53Z" level=warning msg="using host network as the default"
#1 21.05 error: failed to list workers: Unavailable: connection error: desc = "transport: error while dialing: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory"
------
ci-builder
$ docker --config .docker_des_upstream_readonly buildx ls
error: dial unix /run/buildkit/buildkitd.sock: connect: no such file or directory
NAME/NODE     DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
ci-builder    docker-container                             
  ci-builder0 unix:///var/run/docker.sock error            
default *     docker                                       
  default     default                     running 23.0.5   linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
Failed to get status for ci-builder (ci-builder0): listing workers: failed to list workers: DeadlineExceeded: context deadline exceeded

Additional info

There are some duplicate lines in the logs, and those could be caused by Gitlab (it sometimes does that). I am not sure if it is Gitlab or Docker that is the source of the non-printable characters.

My intention with the moby/buildkit:latest image was to get BuildKit 0.11.6, so I was not expecting to find the 0.11.5 BuildKit builder in buildx ls. The 0.11.5 builder is from Docker 23.0.4, I updated to Docker 23.0.5 and rebooted before this.

The sha256:2ab09b027e7f3a0c2e8bb1944ac46de38cebab7145f0bd6effebfe5492c818b6 is my nemesis from moby/buildkit#3809, but I would not expect the cache to be relevant when creating a builder.

All CI machines had docker system prune -af run on them 24 hours ago, so there is not much in the cache, here is docker system df output:

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          34        10        11.69GB   10.09GB (86%)
Containers      10        1         3.006MB   3.006MB (100%)
Local Volumes   152       1         66.6GB    17.41GB (26%)
Build Cache     0         0         0B        0B

I am not sure why it says build cache is 0B, the references mentioned in the panic are from images that are built in the nightly runs.

[pjonsson]

Before hitting this crash, I had #1570 (comment). Did docker --config .ro_config buildx rm ci-builder for the machines with the 0.11.5 builder, and things started working again. I cannot tell if things started working again was because of me removing the 0.11.5 builder, or if it was the same kind of self-healing where leaving the machine for a couple of hours would have been sufficient.