Inline cache not working with dockerd 27.1.1

[tom93]

I'm using docker:27.1.1-dind (latest at the time of writing), and inline cache with --cache-from doesn't work (RUN steps are re-executed instead of being cached). I've bisected this to 26.1.4-dind (good) / 27.0.0-rc.1-dind (bad).

Somewhat similar: #2274.

Steps (using save/load from tar):

# create a docker-in-docker environment (all the following commands run inside this environment)
docker run --rm -it --privileged docker:27.1.1-dind sh -c 'unset DOCKER_HOST; dockerd &>/tmp/log & exec sh'

# create Dockerfile
mkdir app
cat > app/Dockerfile <<"EOF"
FROM busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7
RUN echo this command was not cached | tr a-z A-Z; sleep 3
EOF

# build & save
docker build --cache-to type=inline -t local.example/img app
docker save local.example/img > img.tar

# wipe the local build cache
docker rmi local.example/img
docker system prune -f

# try to build using the loaded image
docker load < img.tar
docker build --cache-from local.example/img -t local.example/img app

The cache isn't used during the second build command. I can tell this because:

• when the second build is running, the message "THIS COMMAND WAS NOT CACHED" is visible for a few seconds (afterwards it's hidden due to the tty progress format)
• when the second build finishes, the line " => [2/2] RUN echo ..." (without the word "CACHED") shows that the RUN step wasn't cached

There are no error messages.

Steps (using a local registry):

# run a local registry
docker run --name registry.example -e REGISTRY_HTTP_ADDR=0.0.0.0:80 registry

# create a docker-in-docker environment
docker run --rm -it --link registry.example --privileged docker:27.1.1-dind sh -c 'unset DOCKER_HOST; dockerd --insecure-registry=registry.example &>/tmp/log & exec sh'

# create Dockerfile
mkdir app
cat > app/Dockerfile <<"EOF"
FROM busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7
RUN echo this command was not cached | tr a-z A-Z; sleep 3
EOF

# build & push
docker build --cache-to type=inline -t registry.example/img app
docker push registry.example/img

# wipe the local build cache (emulates a clean machine)
docker rmi registry.example/img
docker system prune -f

# try to build using the registry cache
docker pull registry.example/img
docker build --cache-from registry.example/img -t registry.example/img app