Auto merge of rust-lang#114763 - Kobzol:fix-ci-docker-caching, r=Mark…

…-Simulacrum

CI: fix Docker layer caching

As reported by `@klensy` on [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/docker.20images.20always.20rebuilded), Github Actions have recently updated their Docker version from 20.x to 23.x, which enabled the BuildKit build backend by default.

This broke our way of performing Docker layer caching on CI, which immediately made all non-PR CI builds (including try builds) ~1 hour longer (Docker caching didn't work on PR builds before, so it wasn't affected). The moment this started happening can be seen [here](https://github.com/rust-lang-ci/rust/actions?page=2&query=branch%3Aauto+is%3Asuccess).

The problem is with the following command:
```
docker history -q rust-ci | \
          grep -v missing | \
          xargs docker save | \
          gzip | \
          $upload
```
which returns the intermediate layers as `<missing>`, if BuildKit is enabled. This was investigated by `@klensy` in rust-lang#114621. Thanks for that!

I will continue experimenting with how we can enable the cache with BuildKit in rust-lang#114762, but for the time being, I think that we should just hotfix this.

This PR reverts the build backend back to the old one, which fixes the caching. However, we also have to bust the cache of all Dockerfiles, otherwise caching would only start kicking in for them the next time they are updated (or the next time GH updates their docker version). Because when the Docker version was updated the last time, the Dockerfiles were cached on S3 with basically an empty cache, and unless we bust it, even after reverting to the old build engine, the CI script would just download the empty cache and rebuild the Dockerfile from scratch, thus nullifying our fix.

r? `@Mark-Simulacrum`

src/ci/docker/run.sh

@@ -63,6 +63,11 @@ if [ -f "$docker_dir/$image/Dockerfile" ]; then
       uname -m >> $hash_key
 
       docker --version >> $hash_key
+
+      # Include cache version. Currently it is needed to bust Docker
+      # cache key after opting in into the old Docker build backend.
+      echo "1" >> $hash_key
+
       cksum=$(sha512sum $hash_key | \
         awk '{print $1}')
 
@@ -90,6 +95,12 @@ if [ -f "$docker_dir/$image/Dockerfile" ]; then
         context="$script_dir"
     fi
     echo "::group::Building docker image for $image"
+
+    # As of August 2023, Github Actions have updated Docker to 23.X,
+    # which uses the BuildKit by default. It currently throws aways all
+    # intermediate layers, which breaks our usage of S3 layer caching.
+    # Therefore we opt-in to the old build backend for now.
+    export DOCKER_BUILDKIT=0
     retry docker \
       build \
       --rm \