buildx write_package or create_package errors with ghcr.io using GITHUB_TOKEN

[tobybellwood]

EDIT: there are a couple of grumbles on the interwebs now, seemingly not just me - https://downdetector.com.au/status/github/

Getting the following error suddenly (this all worked yesterday)

#41 [auth] xxx/ooo:pull,push token for ghcr.io
#41 DONE 0.0s

#39 exporting to image
#39 pushing layers 0.4s done
#39 ERROR: unexpected status: 403 Forbidden
------
 > exporting to image:
------
ERROR: failed to solve: unexpected status: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: unexpected status: 403 Forbidden

adding debug logs to buildx shows the issue

time="2022-09-07T03:54:40Z" level=debug msg="unexpected response" body="{"errors":[{"code":"DENIED","message":"permission_denied: write_package"}]}\n" digest="sha256:8b6a8f742917d3a4b32cd5eac2044e12eefbd97e746780b3447278e010a06b16" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip resp="&{403 Forbidden 403 HTTP/1.1 1 1 map[Content-Length:[76] Content-Type:[application/json] Date:[Wed, 07 Sep 2022 03:54:40 GMT] Docker-Distribution-Api-Version:[registry/2.0] X-Github-Request-Id:[0680:3029:53996:119F6D:63181600]] {0xc0006ebf40} 76 [] false false map[] 0xc000a3df00 0xc0007dd340}" size=135

But - I've followed all the correct configurations - is there a current issue with ghcr that may cause this?

      -
        name: Login to GHCR
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

I can reliably replicate this error across repos, users and packages

[nagimov]

Not sure if related - I'm getting a similar buildx error using actions. Everything worked fine last month using the same workflow yaml

------
 > pushing ghcr.io/nagimov/agendav-docker:2.5.0 with docker:
------
ERROR: denied: permission_denied: write_package
Error: buildx failed with: ERROR: denied: permission_denied: write_package

Job setup shows correct read/write access assigned

GITHUB_TOKEN Permissions
  Contents: read
  Metadata: read
  Packages: write

Login to ghcr via

      - name: login to ${{ env.REGISTRY }}
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

Full build log here, action workflow here.

[tobybellwood]

now there's an incident - https://www.githubstatus.com/incidents/d181frs643d4

[wilkosz]

We are still experiencing this issue

#16 pushing layer b0abfcd12b0f 0.6s done
#16 pushing layer 1af8a44b7f6c 0.6s done
#16 pushing layer 35534433e8ac 0.6s done
#16 pushing layer ad6481d8e9b5 0.6s done
#16 ERROR: denied: permission_denied: write_package
#2 [internal] load .dockerignore
------
 > pushing ghcr.io/random/dochub-base:latest with docker:
------
ERROR: denied: permission_denied: write_package
Error: buildx failed with: ERROR: denied: permission_denied: write_package

[cyclinder]

+1 😄

[qwqVictor]

+1😂.

[tobybellwood]

This was my yesterday… I now know far more about debugging buildx actions than I need to

[smlx]

I just had an image push successfully 🤞

[JoaaoVerona]

Seems to be ok now.

[aakashksunny13]

Looks like the issue is resurfacing intermittently and resolving after some time.
Faced auth issue today as well

#16 ERROR: denied: permission_denied: write_package
------
 > pushing ghcr.io/**** with docker:
------
error: denied: permission_denied: write_package
Error: buildx failed with: error: denied: permission_denied: write_package 

[Yikun]

#34 [auth] apache/apache-spark-ci-image:pull,push token for ghcr.io
#34 DONE 0.0s

#33 exporting to image
#33 pushing layers 0.8s done
#33 pushing manifest for ghcr.io/apache/apache-spark-ci-image:master-3196231785@sha256:5168e7ed98b7c4a27fc836768b831a4d2cd81482baf1904d53584a9ab829dec4
#33 pushing manifest for ghcr.io/apache/apache-spark-ci-image:master-3196231785@sha256:5168e7ed98b7c4a27fc836768b831a4d2cd81482baf1904d53584a9ab829dec4 0.2s done
#33 ERROR: failed commit on ref "manifest-sha256:5168e7ed98b7c4a27fc836768b831a4d2cd81482baf1904d53584a9ab829dec4": unexpected status: 403 Forbidden
------
 > exporting to image:
------
ERROR: failed to solve: failed commit on ref "manifest-sha256:5168e7ed98b7c4a27fc836768b831a4d2cd81482baf1904d53584a9ab829dec4": unexpected status: 403 Forbidden
Error: buildx failed with: ERROR: failed to solve: failed commit on ref "manifest-sha256:5168e7ed98b7c4a27fc836768b831a4d2cd81482baf1904d53584a9ab829dec4": unexpected status: 403 Forbidden

[flexchar]

Also worth checking out the Package Settings as listed here. It helped me out!
https://stackoverflow.com/a/72585915/5883200

[gilacost]

This is still happening 😅

[cherusk]

Happened again, doing everything as reported by others.

[dendrebeden]

Happened again :(

[smlx]

This is still a problem. I just had an image successfully push on one PR, then minutes later I'm getting 403s on PRs in the same repo.

• Successful run
• Failed run (1 min later)

The failing PR is only bumping an image tag 😕

[smlx]

Okay, this one was user error. The failing PR was from Dependabot, which doesn't have permission to push images to the project's gchr.io image registry.

I worked around this problem by not pushing to the image registry on dependabot PRs.

[mondeep-maity]

Facing the issue again, When we are trying to push images to the GHCR we are getting denied: permission_denied: write_package error. Sometimes it's working sometimes it's not.

Is there any issues going on with GitHub Packages? @tobybellwood

[stefanhendriks]

Got the same error : denied: permission_denied: write_package. Using new credentials (recreated a PAT with proper permissions), login works but can't seem to get past the push image step. Uploading to ghcr.io , GH status page says everything is ok.

I did find it odd that only one a specific repository this seems to happen. That repository has very little activity; usually it gets rebuild/build/pushed on minor changes. So it stands out that it suddenly fails to push. Especially since the same credentials are used for another repository that seem to work fine.

Then it occurred to me that within organisation packages you can set permissions; I noticed on the working repository we used "inherited access". (ie, via ORG->Packages->(package name)->Package Settings->(somewhere below)). On the failing repository we had not. I made sure that we also used "inherited access" and made sure the proper team(s)/member(s) got access and voila it worked.

Somehow this got introduced between two publishes for our repo and this broke?

[jaisor]

Had the same issue and wasted hours chasing it. Permissions were specified as expected, everything was supposed to work according to docs. Eventually I gave up on GITHUB_TOKEN and instead I generated a new Personal access token (classic), stored its value in a repository secret and used that instead of GITHUB_TOKEN, it worked right away. Only conclusion is that there is something broken with GITHUB_TOKEN