Auth not successful when using GCP Artifact Registry

[chrissng]

Actual behavior

When building and pushing and image to a repository hosted in GCP Artifact Repository, authentication failed with the error:

E0506 12:21:31.693453       8 metadata.go:248] Failed to unmarshal scopes: json: cannot unmarshal string into Go value of type []string
E0506 12:21:31.698839       8 metadata.go:154] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
E0506 12:21:31.702112       8 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "asia-southeast1-docker.pkg.dev/xxxxx-xxxxx/xxxxx-xxxxx-workload/chrissng2-1e07e4bb:35e5f5f6-1588767684": creating push check transport for asia-southeast1-docker.pkg.dev failed: GET https://asia-southeast1-docker.pkg.dev/v2/token?scope=repository%3Axxxxx-xxxxx%2Fxxxxx-xxxxx-workload%2Fchrissng2-1e07e4bb%3Apush%2Cpull&service=asia-southeast1-docker.pkg.dev: DENIED: Permission "artifactregistry.repositories.uploadArtifacts" denied on resource "projects/xxxxx-xxxxx/locations/asia-southeast1/repositories/xxxxx-xxxxx-workload" (or it may not exist)
E0506 12:21:33.835507       8 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0506 12:21:33.847410       8 metadata.go:248] Failed to unmarshal scopes: json: cannot unmarshal string into Go value of type []string
E0506 12:21:33.851971       8 metadata.go:154] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
E0506 12:21:33.854162       8 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "asia-southeast1-docker.pkg.dev/xxxxx-xxxxx/xxxxx-xxxxx-workload/chrissng2-1e07e4bb:35e5f5f6-1588767684": creating push check transport for asia-southeast1-docker.pkg.dev failed: GET https://asia-southeast1-docker.pkg.dev/v2/token?scope=repository%3Axxxxx-xxxxx%2Fxxxxx-xxxxx-workload%2Fchrissng2-1e07e4bb%3Apush%2Cpull&service=asia-southeast1-docker.pkg.dev: DENIED: Permission "artifactregistry.repositories.uploadArtifacts" denied on resource "projects/xxxxx-xxxxx/locations/asia-southeast1/repositories/xxxxx-xxxxx-workload" (or it may not exist)
E0506 12:21:48.485287       8 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0506 12:21:48.498754       8 metadata.go:248] Failed to unmarshal scopes: json: cannot unmarshal string into Go value of type []string
E0506 12:21:48.503576       8 metadata.go:154] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
E0506 12:21:48.505769       8 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url

Expected behavior

Authentication and pushing of the image should succeed.

To Reproduce
Steps to reproduce the behavior:

1. Set up docker repository in GCP Artifact Registry
2. Setup kaniko job that uses GKE Workload Identity as auth mechanism
3. Set up .docker/config.json to include something like the following:

{
  "auths": {},
  "credHelpers": {
    "asia.gcr.io": "gcr",
    "gcr.io": "gcr",
    "asia-docker.pkg.dev": "gcr",
    "asia-southeast1-docker.pkg.dev": "gcr"
  }
}

Additional Information

• Dockerfile
  https://github.com/GoogleContainerTools/kaniko/blob/v0.21.0/deploy/Dockerfile_debug
• Build Context
  N/A
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor:debug-v0.21.0

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [No]
Please check if the build works in docker but not in kaniko	- [Yes]
Please check if this error is seen when you use --cache flag	- [No]
Please check if your dockerfile is a multistage dockerfile	- [No]