Unable to remove image from GCR registry

[YevheniiSemendiak]

Description:

Got in #2908
In short - it is impossible to remove the container images from GCR. The GCR API has probably changed since our registry API acts as a proxy for the auth process.
We need to investigate the root cause of this issue and where to resolve it (on the API side, or at the client).

traceback

> GET /v2/yevheniisemendiak/alpine/tags/list?n=30 HTTP/1.1
> User-Agent: NeuroCLI/22.7.1 (linux)
> Authorization: Basic ...
> b3: ...
> sentry-trace: 63ef54238a3c7cad39365e761a2e5920-ce64b7ede3d04713-
> 
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 361
< Content-Type: application/json; charset=utf-8
< Date: Fri, 17 Feb 2023 10:17:07 GMT
< docker-distribution-api-version: registry/2.0
< Server: Docker Registry
< x-frame-options: SAMEORIGIN
< x-service-version: platform-registry-api/23.1.1
< x-xss-protection: 0
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< {"child": [], "manifest": {"sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501": {"imageSizeBytes": "3374446", "layerId": "", "mediaType": 
"application/vnd.docker.distribution.manifest.v2+json", "tag": ["latest"], "timeCreatedMs": "1676090802558", "timeUploadedMs": "1676582665070"}}, "name": 
"yevheniisemendiak/alpine", "tags": ["latest"]}
> HEAD /v2/yevheniisemendiak/alpine/manifests/latest HTTP/1.1
> User-Agent: NeuroCLI/22.7.1 (linux)
> Accept: application/vnd.docker.distribution.manifest.v2+json
> Authorization: ...
> b3: ...
> sentry-trace: 63ef54238a3c7cad39365e761a2e5920-f4d27936dd4e0135-
> 
< HTTP/1.1 200 OK
< Content-Length: 528
< Content-Type: application/vnd.docker.distribution.manifest.v2+json
< Date: Fri, 17 Feb 2023 10:17:08 GMT
< docker-content-digest: sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501
< docker-distribution-api-version: registry/2.0
< Server: Docker Registry
< x-frame-options: SAMEORIGIN
< x-service-version: platform-registry-api/23.1.1
< x-xss-protection: 0
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Deleting image://default/yevheniisemendiak/alpine:latest identified by sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501
> GET /v2/yevheniisemendiak/alpine/tags/list?n=30 HTTP/1.1
> User-Agent: NeuroCLI/22.7.1 (linux)
> Authorization: Basic ...
> b3: ...
> sentry-trace: 63ef54238a3c7cad39365e761a2e5920-edbaf1a9c3e91c59-
> 
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Length: 361
< Content-Type: application/json; charset=utf-8
< Date: Fri, 17 Feb 2023 10:17:08 GMT
< docker-distribution-api-version: registry/2.0
< Server: Docker Registry
< x-frame-options: SAMEORIGIN
< x-service-version: platform-registry-api/23.1.1
< x-xss-protection: 0
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< {"child": [], "manifest": {"sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501": {"imageSizeBytes": "3374446", "layerId": "", "mediaType": 
"application/vnd.docker.distribution.manifest.v2+json", "tag": ["latest"], "timeCreatedMs": "1676090802558", "timeUploadedMs": "1676582665070"}}, "name": 
"yevheniisemendiak/alpine", "tags": ["latest"]}
> DELETE /v2/yevheniisemendiak/alpine/manifests/sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501 HTTP/1.1
> User-Agent: NeuroCLI/22.7.1 (linux)
> Authorization: Basic ...
> b3: ...
> sentry-trace: 63ef54238a3c7cad39365e761a2e5920-fa4b6333d6eaa9ef-
> 
< HTTP/1.1 400 Bad Request
< Cache-Control: private
< Content-Type: application/json
< Date: Fri, 17 Feb 2023 10:17:09 GMT
< docker-distribution-api-version: registry/2.0
< Server: Docker Registry
< x-frame-options: SAMEORIGIN
< x-service-version: platform-registry-api/23.1.1
< x-xss-protection: 0
< Via: 1.1 google
< Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
< Transfer-Encoding: chunked
< {"errors":[{"code":"GOOGLE_MANIFEST_DANGLING_TAG","message":"Manifest is still referenced by tag: latest"}]}

Example:

Steps to reproduce the behavior:

1. Create (push) image to private GCR registry.
2. Try to remove the image (by using -f flag for CLI).
3. Get error:
   ERROR: Illegal argument(s) ({"errors":[{"code":"GOOGLE_MANIFEST_DANGLING_TAG","message":"Manifest is still referenced by tag: latest"}]})

Additional context:

Neuro Platform Client 22.7.1
platform-registry-api/23.1.1