Add s3 cache

Largely inspired from moby#2323

Signed-off-by: Bertrand Paquet <[email protected]>

README.md

@@ -58,6 +58,7 @@ You don't need to read this document unless you want to use the full-featured st
     - [Registry (push image and cache separately)](#registry-push-image-and-cache-separately)
     - [Local directory](#local-directory-1)
     - [GitHub Actions cache (experimental)](#github-actions-cache-experimental)
+    - [S3 cache (experimental)](#s3-cache-experimental)
   - [Consistent hashing](#consistent-hashing)
 - [Metadata](#metadata)
 - [Systemd socket activation](#systemd-socket-activation)
@@ -425,6 +426,46 @@ in your workflow to expose the runtime.
 * `type=gha`
 * `scope=buildkit`: which scope cache object belongs to (default `buildkit`)
 
+#### S3 cache (experimental)
+
+```bash
+buildctl build ... \
+  --output type=image,name=docker.io/username/image,push=true \
+  --export-cache type=s3 \
+  --import-cache type=s3
+```
+
+The following attributes are required:
+* `bucket`: AWS S3 bucket (default `$AWS_BUCKET`)
+* `region`: AWS region (default `$AWS_REGION`)
+
+Storage locations:
+* blobs: `s3://<bucket>/<prefix><blobs_prefix>/<sha256>`, default: `s3://<bucket>/blobs/<sha256>`
+* manifests: `s3://<bucket>/<prefix><manifests_prefix>/<name>`, default: `s3://<bucket>/manifests/<name>`
+
+The following attributes are required and will be automatically set when using [iam-roles-for-service-accounts](https://github.com/awsdocs/amazon-eks-user-guide/blob/master/doc_source/iam-roles-for-service-accounts-technical-overview.md).
+* `role`: AWS role (default `$AWS_ROLE_ARN`)
+* `session`: AWS assume role session name (default `$AWS_ROLE_SESSION_NAME`)
+* `token`: AWS web identity token file (default `$AWS_WEB_IDENTITY_TOKEN_FILE`)
+
+`--export-cache` options:
+* `type=s3`
+* `mode=min` (default): only export layers for the resulting image
+* `mode=max`: export all the layers of all intermediate steps.
+* `prefix`: global prefix to store / read files on s3. Default: empty
+* `blobs_prefix`: global prefix to store / read blobs on s3. (default: `blobs/`)
+* `manifests_prefix`: global prefix to store / read blobs on s3. (default: `manifests/`)
+* `name=buildkit`: name of the manifest to use (default `buildkit`)
+* `duplicates`: list of names on which we want to duplicate the manifest, separated by `;`. The standard use case is to use the
+  git sha1 as name, and the branch name as duplicate, and load both in `import-cache`
+
+`--import-cache` options:
+* `type=s3`
+* `prefix=`: global prefix to store / read files on s3. Default: empty
+* `blobs_prefix=`: global prefix to store / read blobs on s3. (default: `blobs/`)
+* `manifests_prefix=`: global prefix to store / read blobs on s3. (default: `manifests/`)
+* `name=buildkit`: name of the manifest to use (default `buildkit`)
+
 ### Consistent hashing
 
 If you have multiple BuildKit daemon instances but you don't want to use registry for sharing cache across the cluster,