(lambda): lambda.DockerImageFunction can not be used alongside containerd to utilize cache storage backends

[neilferreira]

Describe the bug

Context:

CDK provides the ability to utilize Docker cache storage backends as per the docs.

In order to make use of cache storage backends such as a local directory or external registry, it is a requirement to enable containerd image store in Docker.

Once containerd has been enabled, you are able to populate the build cache parameters within CDK, and subsequent builds on different CI runners would all utilise the same backend cache, drastically improving the CDK build and deployment process.

Bug

Once containerd has been enabled, any Lambda function utilizing the Docker runtime fails to deploy.

Lambda code:

new lambda.DockerImageFunction(
  this,
  "LambdaFn",
  {
    functionName: "containerd-function",
    code: lambda.DockerImageCode.fromImageAsset(__dirname + "/lambda"),
  }
);

Errror message:

UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "The image manifest or layer media type for the source image ... is not supported. (Service: Lambda, Status Code: 400, Request ID: ...)"

A likely resolution:

As per a similar issue noted on #30258 - The only workaround for this is to currently use BUILDX_NO_DEFAULT_ATTESTATIONS=1 cdk deploy

It would be better if CDK explicitly adds --provenance=false in its calls to docker buildx, or provides the ability to do so.

See https://docs.docker.com/reference/cli/docker/buildx/build/#provenance and https://docs.docker.com/build/attestations/attestation-storage/

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

n/a

Expected Behavior

I would expect to be able to utilize the documented cache backends feature within CDK when deploying Lambda functions that utilises a container image. These two CDK features are not able to work together.

Current Behavior

Lambda fails to create or update the Lambda function with the following error message:

The image manifest or layer media type for the source image is not supported

Reproduction Steps

1. Enable containerd in Docker
2. Create a Lambda function in CDK using:

new lambda.DockerImageFunction(
  this,
  "LambdaFn",
  {
    functionName: "containerd-function",
    code: lambda.DockerImageCode.fromImageAsset(__dirname + "/lambda"),
  }
);

3. Create a Dockerfile using:

FROM --platform=linux/amd64 public.ecr.aws/lambda/python:3.12

COPY handler.py ./

CMD [ "handler.handler" ]

4. Create a handler.py using:

def handler(event, context):
    return {"success": True}

Run cdk deploy to deploy the function.

Possible Solution

Prefixing BUILDX_NO_DEFAULT_ATTESTATIONS=1 to cdk deploy works around this issue.

Additional Information/Context

No response

CDK CLI Version

2.160.0 (build 7a8ae02)

Framework Version

No response

Node.js Version

v18.17.1

OS

MacOS 14.6

Language

TypeScript

Language Version

No response

Other information

No response

[khushail]

Hi @neilferreira , thanks for reaching out.

I am able to repro the scenario with the given instructions and code. This is the error (UnsupportedImageLayerDetected) seen while deployment -

Since this has a workaround, I am marking this as P2 which means it won't be immediately addressed by the team but would be in their radar.

I will bring this up to the team for their input if this is something we are actively investigating and share our thoughts here if possible.

Thanks.

[HBobertz]

Hi @neilferreira , thanks for reaching out.

I am able to repro the scenario with the given instructions and code. This is the error (UnsupportedImageLayerDetected) seen while deployment -

Since this has a workaround, I am marking this as P2 which means it won't be immediately addressed by the team but would be in their radar.

I will bring this up to the team for their input if this is something we are actively investigating and share our thoughts here if possible.

Thanks.

appreciate the replication. Yeah this seems to just be a bug on our end but given we have a known workaround I think p2 is appropriate. Keeping this in the backlog for now

[blimmer]

This error is pretty annoying to recover from, as well, because you have to invalidate the DockerImageAsset to try pushing again. I think it might be worthy of a p1.

[jbiggar]

For potentially relevant context from the Docker Desktop docs:

The containerd image store is enabled by default in Docker Desktop version 4.34 and later

4.34 was released in August 2024.

[rhjsong]

Can confirm this error occurs for Docker Desktop starting v4.34.

BUILDX_NO_DEFAULT_ATTESTATIONS=1 cdk deploy did not solve the issue. Downgrading Docker Desktop to before v4.34 solved it.

[phuhung273]

Fix waiting for dependency cdklabs/cloud-assembly-schema#102

[moyarich]

I had to do all the steps mentioned here and more to get my CDK lambda function to deploy. So I outlined the steps to address the common issues when pushing Docker images to Amazon Elastic Container Registry (ECR), especially for AWS CDK deployments using DockerImageFunction.

I also provided code examples : https://github.com/moyarich/how-to-resolve-amazon-ecr-push-issues

TL;DR - Steps to Fix ECR Issues

Here's the summary:

Common Errors Addressed

1. 400 Bad Request Errors - Manifest Issues
2. Lambda Stabilization Failures - Invalid Image

---

Steps to Resolve ECR Issues

1. Enable Docker BuildKit

   • Option 1 : Use BuildKit for enhanced builds:

     export DOCKER_BUILDKIT=1

   • Option 2 : set Docker Build as a passthrough to Buildx by creating an alias:

     alias 'docker build'='docker buildx build'

---

2. Clean Up ECR Repositories
   • Prune unused local images:

     docker system prune --all

   • Delete 0-byte images in ECR using the AWS console or the AWS CLI .
   • I provided the code for a shell script that automatically delete these empty images from the repository using the AWS CLI.

---

3. Force a New Tag in ECR
   • Update the Docker build context(source code)
   • or modify the Dockerfile.
   • or Change the logical ID of the DockerImageFunction in the CDK stack to generate new tags.

---

4. Disable Metadata Attestations
   • Set BUILDX_NO_DEFAULT_ATTESTATIONS=1
   • or include --provenance=false and --sbom=false options when building the image.

---

5. Automate Deployment
   This is a common problem for pushing images created by DockerImageFunction to the immutable repository create by cdk bootstrap, so I automated the important bits via package.json

   • Use package.json scripts to streamline deployment, including ECR login and enabling BuildKit:

{
  "scripts": {
    "deploy": "DOCKER_BUILDKIT=1 sh -c 'cd my_cdk_stacks && npm run cdk deploy -- -c stack_name=\"${STACK_NAME:-}\" --all --require-approval never'",
    "predeploy": "aws ecr get-login-password | docker login --username AWS --password-stdin $(aws sts get-caller-identity --query 'Account' --output text).dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com",
    "destroy": "cd my_cdk_stacks && npm run cdk destroy -- -c stack_name=\"${STACK_NAME:-}\" --all"
  }
}

[StarfieldDesigns]

Hey there, I had this issue as well, my specs:

• CPU = Mac Mini m4
• OS = Sequoia 15.3
• Docker Desktop version = v.4.38.0
• CDK version = 2.178.1 (build ae342cb)

Followed the steps for the workaround described in this thread which did get my CDK deploy working again, thanks to all who contributed to that. However, the workaround is a bit of lengthy process so would be great if CDK Dev teams were able to bump this issue to P1 and patch asap (: I understand you guys probably have a million things to do, and this seems to be a somewhat uncommon problem, but would love to see a patch sooner rather than later. Thanks all!

[hervenivon]

#31548 (comment)

I found the following workaround for me (Mac Mini m4, Sequoia 15.3, Docker Desktop version = v4.38.0, cdk 2.177.0).

1. Navigate to Settings in Docker Desktop.
2. In the General tab, uncheck Use containerd for pulling and storing images.
3. Select Apply & Restart.
4. Clean all: docker system prune -af --volumes && docker builder prune -af
5. On Amazon ECR for the given environment, delete the images with the 0 size manifest.

No CDK code change