accessing secret very slow

[pongnguy]

Description

Any step that uses a build secret is very slow. Should be fast since file is local and of a very small size.

Reproduce

docker build --secret id=secret-key,src=secret-source -t image-name .

Expected behavior

Step involving key should take the same amount of time as other steps of similar complexity

docker version

Client: Docker Engine - Community
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:17:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:17:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client: Docker Engine - Community
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.25.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 3
  Paused: 0
  Stopped: 2
 Images: 39
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: active
  NodeID: nrfo0xui5r0hhpj5zzkafjjqd
  Is Manager: true
  ClusterID: uh2qrtf90k9xa3yimp07wsu60
  Managers: 2
  Nodes: 2
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.0.0.151
  Manager Addresses:
   10.0.0.151:2377
   10.0.0.248:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-107-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 40
 Total Memory: 62.78GiB
 Name: r73043420
 ID: d9387ddf-63d1-4828-a1dc-b60c6bc81f4a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running Swarm in a two-manager configuration. This configuration provides
         no fault tolerance, and poses a high risk to lose control over the cluster.
         Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
         Swarm for fault-tolerance.

Additional Info

No response

[thaJeztah]

Let me transfer this to the build repository, which is where the CLI side of build is maintained

[tonistiigi]

Can you put together a reproducer?

[laurence-kobold]

I'm also seeing this issue. Here is a reproducer:

FROM python:latest

RUN echo "without mount"

RUN --mount=type=secret,id=SECRET,uid=1000 \
    echo "with mount"

The second command takes ~5s to start running.

I've seen run commands take more than a minute to start running when mounting secrets during the build of an image. In this case I'm building a much larger image, so I'm wondering if this time scales with the size of the image

[rezo1134]

We have also seen that versions starting with 26.0.0 have increased build times when mounting secrets. Builds are very fast up until the mount step, and then it just freezes for a couple minutes then eventually continues. We downgraded to Docker version <26 (25.0.5 atm) and builds are fast again. So far we've confirmed even the most recent Version 27 upgrade suffers from the same issue.

[mongi3]

We've observed the same slow secret behavior in our builds starting with 26.0.0 and have similarly downgraded to 25.0.5 to maintain build performance. Observed slowness can be in the minutes for steps involving mounting of secrets, like something is hung pending a timeout (pure conjecture).

Ticket is tagged needs more info. What more is needed?

[crazy-max]

I will take a look, is someone able to repro with a container builder as well?

[prhbrt]

This seems fixed in buildkit 0.15.2 and then in builx v0.17.0-rc1.

For noobs in docker dev, when will this propagate to docker compose? I guess the current newest release still uses buildx 0.16.1 and hence will still have the lag?