CircleCI is failing on test_backend, because /cache/docker.tar is not restored

[jwhitlock]

After merging PR #1624, builds are failing on the test_backend job, on the "Restore Docker image" step:

docker load --input /cache/docker.tar
open /cache/docker.tar: no such file or directory
Exited with code exit status 1

Working backwards, the step "Restoring cache" is failing to load the file:

Found a cache from build 9011 at v1-backend-build-main
Size: 508 MiB
Cached paths:
  * /cache/docker.tar

Downloading cache archive...
Unarchiving cache...
Skipping writing "cache/docker.tar" - open /cache/docker.tar: no such file or directory

This is created by the build_backend job, in the "Saving cache" step:

Creating cache archive...
Uploading cache archive...
Stored Cache to v1-backend-build-main-1648139534
  * /cache/docker.tar
Total size uploaded: 508 MiB

The CircleCI cache documentation suggests that the cache has size limits:

We recommend keeping cache sizes under 500MB. This is our upper limit for corruption checks. Above this limit, check times would be excessively long. You can view the cache size from the CircleCI Jobs page within the restore_cache step. Larger cache sizes are allowed, but may cause problems due to a higher chance of decompression issues and corruption during download. To keep cache sizes down, consider splitting them into multiple distinct caches.

It is possible that we're seeing a transitory issue.

[jwhitlock]

I've been working on this issue, but didn't file it first. Some attempts:

• PR Revert "Update to Python 3.9.10" #1683 - This did not fix the problem, even though the image was returned to the old size (498 MiB, down from 508MiB)
• PR CircleCI fix: Split docker.tar into 100 MB parts #1684 - This appears to work, and included changing cache keys.
• PR CircleCI: Switch to alpine #1685 - This image has a much smaller base image (5 MB), but requires installing several build tools that may expand the size

I'll probably return to this in the morning.