Kaniko can't pull public images from gcr.io

[filesnate]

Actual behavior
Using kaniko to pull and build a kaniko image fails (kaniko is a public image) because it requires credentials.

Expected behavior
I expect kaniko to pull a public image w/out requiring credentials.

To Reproduce
Steps to reproduce the behavior:

1. Dockerfile is

FROM gcr.io/kaniko-project/executor:debug

# Over-ride the default entrypoint to avoid always calling /kaniko/executor
# so we can have additional build steps.
ENTRYPOINT [""]

2. Build image

% docker run -v `pwd`:/workspace gcr.io/kaniko-project/executor:debug --context=/workspace --no-push
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
ERRO[0000] Error while retrieving image from cache: gcr.io/kaniko-project/executor:debug error getting credentials - err: exit status 1, out: `docker-credential-gcr/helper: could not retrieve GCR's access token: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.` 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
error building image: error getting credentials - err: exit status 1, out: `docker-credential-gcr/helper: could not retrieve GCR's access token: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.`
%

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use --cache flag	Yes
Please check if your dockerfile is a multistage dockerfile	No

[tejal29]

@filesnate Thank you for opening this issue. Kaniko uses gooogle/go-container-registry to retrieve images.
From your docker run command looks like you haven't volume mounted docker credentials.
See google/go-containerregistry#666 (comment) for why this could be the issue.

Can you confirm again ?
/cc @samos123

Meanwhile, we can look into if executor:debug image creates a docker config.

[filesnate]

@tejal29 - I shouldn't need any credentials to pull a publically available image.

% docker pull gcr.io/kaniko-project/executor:debug
...
Status: Image is up to date for gcr.io/kaniko-project/executor:debug
gcr.io/kaniko-project/executor:debug
%

No credentials are necessary to fetch that image.

If I change the image in the Dockerfile to a different publicly available image, it works.

Dockerfile:

FROM alpine:3.9.5

Now, the command runs fine.

% docker run -v `pwd`:/workspace gcr.io/kaniko-project/executor:debug --context=/workspace --no-push
INFO[0000] Resolved base name alpine:3.9.5 to alpine:3.9.5 
INFO[0000] Resolved base name alpine:3.9.5 to alpine:3.9.5 
INFO[0000] Retrieving image manifest alpine:3.9.5       
INFO[0001] Retrieving image manifest alpine:3.9.5       
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest alpine:3.9.5       
INFO[0003] Retrieving image manifest alpine:3.9.5       
INFO[0004] Skipping unpacking as no commands require it. 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Resolving paths                              
INFO[0004] Skipping push to container registry due to --no-push flag 
%

I suspect that having an image name start with gcr.io triggers behavior where kaniko (incorrectly) assumes credentials are necessary.

[filesnate]

Also, while I do have a file ~/.docker/config.json, it has no entries for gcr.io.

[filesnate]

In the kaniko image, there exists the file /kaniko/.docker/config.json with the following contents.

/kaniko/.docker # more config.json 
{
        "auths": {},
        "credHelpers": {
                "asia.gcr.io": "gcr",
                "eu.gcr.io": "gcr",
                "gcr.io": "gcr",
                "staging-k8s.gcr.io": "gcr",
                "us.gcr.io": "gcr"
        }
}

If this file is replaced with a blank json file contents of {}, then kaniko works properly. This default configuration files is causing issues with any registry listed above, as it causes them to REQUIRE credentials that are not setup in the default case. IMO, the contents of config.json should be replaced with a blank json file.

Dockerfile:

FROM gcr.io/kaniko-project/executor:debug

config.json:

{}

% docker run -v `pwd`/config.json:/kaniko/.docker/config.json:ro -v `pwd`:/workspace gcr.io/kaniko-project/executor:debug --context=/workspace --no-push
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Resolved base name gcr.io/kaniko-project/executor:debug to gcr.io/kaniko-project/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0002] Retrieving image manifest gcr.io/kaniko-project/executor:debug 
INFO[0002] Skipping unpacking as no commands require it. 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Resolving paths                              
INFO[0002] Skipping push to container registry due to --no-push flag

[tejal29]

Thanks a lot @filesnate for debugging this.
I agree, we should replace config.json with empty file and other users who use private images can add their config as a volume mount inside cluster like this
or on the docker run command like you mentioned

Would you up for submitting a PR to remove credentials helper in from the executor image ?

[filesnate]

Let me see what I can do

[filesnate]

#1125

[filesnate]

3rd times a charm. I was rushing to get it done, and didn't look closely at the Dockerfile to see how it was being built.

[filesnate]

@tejal29 - Is there anything I need to do at this point to move this forward? PR is submitted and is passing.

[filesnate]

Thank you!