Possible incompatibility with AWS ECR #826
Comments
|
Do you have a link to your repo? Can you also post the BuildKit logs please (see https://docs.docker.com/build/ci/github-actions/configure-builder/#buildkit-container-logs)? |
The repo is private and just a fresh new for testing this action. I can send you the logs later today. Thanks! |
|
Hi @crazy-max, here are the logs setting Thank you! |
|
Hum I wonder if this is related to an issue with ECR and provenance. Can you try with provenance disabled?: - name: Build and push container image
if: github.event_name == 'push'
uses: docker/build-push-action@v4
with:
context: .
push: true
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
provenance: false |
|
The error persists setting |
|
Hum that's odd. I wonder if this is linked to the build-push-action/.github/workflows/e2e.yml Lines 64 to 70 in 35434f5 And looks good: https://github.com/docker/build-push-action/actions/runs/4312754710/jobs/7523704245#step:11:318 I see you're using the GitHub OIDC provider as shown in https://github.com/aws-actions/configure-aws-credentials#credentials. Can you replace the I see this workflow using this action and looks to work for them: https://github.com/nhost/hasura-auth/actions/runs/4205620537/jobs/7298746334#step:12:913 Also |
|
That's weird... we tried yesterday with the The thing is that this workflow was executed 2 weeks ago, maybe there is something broken now. I'm going to try this: - name: Configure AWS
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.IAM_ROLE_ARN }}
role-session-name: test
- name: Login to Amazon ECR
uses: aws-actions/amazon-ecr-login@v1
- name: Build and publish to Docker Hub and AWS ECR
uses: docker/build-push-action@v3
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
push: true |
|
Ok, the following is super weird. If I use the following I get a - name: Configure AWS
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.IAM_ROLE_ARN }}
role-session-name: test
- name: Login to Amazon ECR
uses: aws-actions/amazon-ecr-login@v1
- name: Build and publish to Docker Hub and AWS ECR
uses: docker/build-push-action@v3
with:
context: .
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
push: true
But with the following I can push the images without any problem: jobs:
build-push-connector-container:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.IAM_ROLE_ARN }}
role-session-name: test
- name: Login to ECR
uses: docker/login-action@v2
with:
registry: ${{ env.ECR }}
# - name: Login to Amazon ECR
# id: login-ecr
# uses: aws-actions/amazon-ecr-login@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
buildkitd-flags: --debug
- name: Build and push container latest image
uses: docker/build-push-action@v4
with:
push: false
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
outputs: type=docker
- name: Push manually
run: docker push ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}Do you have any idea of what to test? Definitely it is not a permissions issue whatsoever. |
I am experiencing the same thing with OCI-compliant (provenance:true) multiplatform images on self-hosted runners. And interestingly, not on all images (only really large ones?). Technically I can work around it by using So for now Here's a gist I made for context. What fails (same issue as yours) The type=oci + import + push workaround Mind you this isn't even using the action. So it seems to be a buildx bug (or maybe some shortcoming on the ECR side?), rather than a EDIT: Your workaround won't work for multiarch OCI-compliant images (see: docker/roadmap#371) ...docker exporter does not currently support exporting manifest listsSECOND EDIT (24/03/23): wanted to follow up on this: A good workaround is to do |
|
In my case the it got working after restart the buildx builder container and re-login in ECR. Logs of the worker |
|
Have the same issue: thx for workaround #826 (comment) |
|
Maybe there is something more going on for other people, but for me it turns out I was simply absent-minded and hadn't referenced the ECR as a resource on the IAM policy. I would triple-check your policies on the IAM role/user that you are using for the action to make sure it's pointed at the correct ECR and that it has all the policies necessary to log into and push to it. Anyway my solution ended up looking like:
I didn't need anything further such as the AWS credentials configure or the ECR login actions. |
Looking at the workflow working for you, you're loading the image to Docker ( - name: Build and push container latest image
uses: docker/build-push-action@v4
with:
load: true
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}
And you're pushing manually in the last step. What's your use case for pushing manually btw? If it works when pushing manually, it should work just fine with the action: jobs:
build-push-connector-container:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-region: ${{ env.AWS_REGION }}
role-to-assume: ${{ env.IAM_ROLE_ARN }}
role-session-name: test
- name: Login to ECR
uses: docker/login-action@v2
with:
registry: ${{ env.ECR }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
with:
buildkitd-flags: --debug
- name: Build and push container latest image
uses: docker/build-push-action@v4
with:
context: .
push: true
file: ${{ env.DOCKERFILE_PATH }}
tags: ${{ env.ECR }}/${{ env.IMAGE_NAME }}:${{ env.TAG }}What's odd is our e2e tests when pushing to ECR or ECR Public look good: https://github.com/docker/build-push-action/actions/runs/4637273377/jobs/8206003788#step:11:306 |
We're pushing mannually since if we use the
That's the thing we've been talking through this thread, there is some issue using the action since we receive a |
|
We’re not currently using OIDC, but I was able to get my Docker Publish to ECR working last night: jobs:
publish:
name: Publish
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- id: aws-credentials
uses: aws-actions/[email protected]
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- id: login-ecr
uses: aws-actions/[email protected]
- id: meta
uses: docker/[email protected]
with:
images: |
${{ steps.login-ecr.outputs.registry }}/${{ env.ECR_REPOSITORY }}
tags: |
type=raw,value=${{ env.RELEASE_TAG }},enable=true
type=ref,event=tag
- uses: docker/[email protected]
- name: Build image and push to ECR
id: docker_build
uses: docker/[email protected]
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
provenance: false
cache-from: type=gha
cache-to: type=gha,mode=maxI wasn’t able to get it working either with this or with the manual push, but it boiled down to a typo in the value for |
|
Thanks for that @halostatue, I'll test it again pushing using the action. Thank you too @crazy-max. |
|
I'm still experiencing this issue, all actions are latest version. Commenting to highlight @halostatue is using user credentials (access and secret key), not OIDC ( |
|
While doing additional testing on the setup with AWS I noticed that if I updated the polices to allow the OIDC policy to perform We finally got it working using OIDC with this policy (some retractions made): Note that we added the following compared to a policy that already worked for regular |
|
I can confirm adding these two actions to the policy attached to the role we are assuming fixed the issue for us. |
|
In my case it was a full ECR. After I removed a number of images from ECR, it started to work again. Then I added a lifecycle policy to the registry :) |
Troubleshooting
Before submitting a bug report please read the Troubleshooting doc.
Behaviour
Steps to reproduce this issue
Assuming you have the right permissions to push to an AWS ECR (in our case we do)
Expected behaviour
The
docker/build-push-action@v4should be able to upload the container image to the container registry.Actual behaviour
The
docker/build-push-action@v4returns a403 Forbiddenerror even when the action has the right credentials to push to the repository. That's because settingpush: falseand using a separatedocker pushresults in the image correctly pushed to the container registry.Configuration
Logs
The text was updated successfully, but these errors were encountered: