Release v1.8.0

[imjasonh]

v1.7.0 was cut in October, then rolled back due to an issue with GCR auth -- the tag wasn't removed, but :latest was changed to point to :v1.6.0.

Issues: #1821 #1805 #1791 #1786

Since #1856 I believe these auth issues should be resolved, and I think it would be a good idea to cut a :v1.8.0 to get this change and others since v1.6.0 out to users.

It looks like there's also #1789, which might be a regression in v1.7.0, which I can take a look at before v1.8.0.

@tejal29 @briandealwis wdyt? I don't have the necessary permissions to rollback the tag in GCR manually, only via GitHub Actions release workflows, so I'd love to have your go-ahead before proceeding.

[imjasonh]

It looks like there's also #1789, which might be a regression in v1.7.0, which I can take a look at before v1.8.0.

nvm, that was fixed by reverting in #1794

[Phylu]

I'd really love to see this new release. In my case, the 1.6 version does not work for images with large layers which is fixed in 1.7 with #1680. However the 1.7 version does not work at all due to #1721 which should be fixed in ee31dc9 which is on the main branch, but still unreleased.

[tejal29]

Thanks @imjasonh for bringing this up. I am OOO. Will check if someone can help with the release.

[imjasonh]

Thanks @tejal29!

I think #970 (comment) might be an issue we should resolve before cutting a release. I don't personally have much context on it but maybe someone else does.

[kun-lu20]

Hi @imjasonh , hope all is well.

Do you have any updates reg releasing v1.8.0? Thank you very much!

[imjasonh]

I believe we're ready to go (someone please let me know if there are blocking issues!).

Just waiting on @tejal29 or someone else from Google's side to give us the go-ahead, and to be around in case we need to do an emergency rollback like we did for v1.6.0->v1.7.0.

In the meantime it'd be really useful to get feedback on the pre-release commit-tagged images (:76624697df879f7c3e3348f22b8c986071af4835, corresponding to commit 7662469, is the newest), so please try those out and let me know if there are any issues!

@briandealwis any thoughts?

[hermanbanken]

Just tried running Kaniko (latest, 1.6.0) with an updated docker-credential-gcr (2.1.0), and it still produces the error of #1786. So the issue might be caused by how the new docker-credential-gcr behaves.

[hermanbanken]

I tried 7662469 and still have issue #1889.

[imjasonh]

I have another question about the release process, which we should clear up before cutting a new release: how is the gcr.io/kaniko-project/executor image produced?

There's GitHub Actions config in place to build and push the image on pushes to the repo, and to tag :latest when tags are pushed. There's also Google Cloud Build cloudbuild.yaml files in https://github.com/GoogleContainerTools/kaniko/tree/main/deploy that seem related to a release -- are there GCB triggers set up for tag changes?

I think we should make sure we won't accidentally release the image twice (and possibly subtly differently), and I'd like to suggest that we use the GitHub Actions flows instead of GCB -- they're more publicly auditable, logs are public, and it's already how we're pushing commit-tagged images.

If the GCB configs aren't used, we can just delete them.

@tejal29 @briandealwis do either of you know?

[henrik242]

What about #1791, is a fix for that planned for 1.8.0?

[imjasonh]

What about #1791, is a fix for that planned for 1.8.0?

I believe that's been fixed by all the auth refactoring since v1.7.0, but as always, testing with a commit-tagged image to verify would be helpful!

[kun-lu20]

Hi @imjasonh @tejal29 , do you have any updates reg the release date of v1.8.0? Thanks very much!

[karanthukral]

Wondering if there is an expected release timeframe for 1.8.0. Running into #1789 on 1.7.0 but head seems to be fine (saw that the change was reverted). Thanks :D

[wlynch]

Hi everyone!

Just want to check in to let everyone know we're looking into cutting a new release soon. 🙏 I'm new to the project so I'll need a bit of time to ramp up, but I'm looking to make sure this happens by ~early March.

[imjasonh]

This is done now! https://github.com/GoogleContainerTools/kaniko/releases/tag/v1.8.0 🎉

Thanks @chuangw6

[tnaroska]

Hi @imjasonh ,
looks like the debug image is not available in gcr, yet:

$ docker pull gcr.io/kaniko-project/executor:v1.8.0-debug
Error response from daemon: manifest for gcr.io/kaniko-project/executor:v1.8.0-debug not found: manifest unknown: Failed to fetch "v1.8.0-debug" from request "/v2/kaniko-project/executor/manifests/v1.8.0-debug".

[imjasonh]

gcr.io/kaniko-project/executor:v1.8.0-debug

Indeed! I'm also not seeing that image

The Build Images workflow should have done it, but seems to have tagged it as :v1.8.0 instead: https://github.com/GoogleContainerTools/kaniko/runs/5474025092?check_suite_focus=true#step:11:1

2022/03/09 02:33:24 Copying from gcr.io/kaniko-project/executor@sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f to gcr.io/kaniko-project/executor:v1.8.0
...
2022/03/09 02:33:27 Copying from gcr.io/kaniko-project/executor@sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f to gcr.io/kaniko-project/executor:

This was the first time our actual release process was exercised, so it seems there are some bugs: it should tag the image as :debug and as :v1.8.0-debug

Someone with GCR permissions should be able to fix this with:

crane cp gcr.io/kaniko-project/executor@sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f gcr.io/kaniko-project/executor:debug

crane cp gcr.io/kaniko-project/executor@sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f gcr.io/kaniko-project/executor:v1.8.0-debug

And we should fix whatever bug in the GitHub Actions workflow caused this.

[imjasonh]

cc @wlynch @chuangw6 ☝️

[chuangw6]

@tnaroska Thanks for flagging this!
I've manually added debug and v1.8.0-debug tag to that image as @imjasonh suggested. Thanks @imjasonh!

Please try again with one of the following commands.

• docker pull gcr.io/kaniko-project/executor:debug
• docker pull gcr.io/kaniko-project/executor:v1.8.0-debug

[gabClark]

I am afraid #1821 is still triggering when trying to upgrade to tag v1.8.0-debug

[FranAguiar]

I'm facing this issue with 1.8.0-debug and with 1.8.1-debug

/kaniko/executor --dockerfile context/Dockerfile --context context 
--destination eu.gcr.io/gcp-project/my-client/develop:latest 
--destination eu.gcr.io/gcp-project/my-client/develop:build-209

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "eu.gcr.io/gcp-project/my-client/develop:latest": creating push check transport for eu.gcr.io failed: GET https://eu.gcr.io/v2/token?scope=repository%3Agcp-project%2Fmy-client%2Fdevelop%3Apush%2Cpull&service=eu.gcr.io: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

I double checked the service account permissions many many times, storage admin and google container agent. What else do I need? I'm a little bit lost here...

Even if I go to 1.6 or 1.5 I got the same error, clearly the error is on my side, but I'm totally lost.

[Ghostwritten]

create secret

kubectl --namespace=default create secret   generic  harbor-regcred --from-file=.dockerconfigjson=config.json --type=kubernetes.io/dockerconfigjson

kubectl get secret harbor-regcred --output="jsonpath={.dat
a.\.dockerconfigjson}" | base64 -d

{
        "auths": {
                "https://harbor.fumai.com/": {
                        "username": "admin",
                        "password": "Harbor12345",
                        "email": "[email protected]",
                        "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                }
        }
}

---
apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: 326takeda/kaniko-project_executor:v1.8.1-debug
    args: ["--dockerfile=/workspace/Dockerfile",
            "--context=dir://workspace",
            "--destination=harbor.fumai.com/library/devops-toolkit:1.0.0"]
    volumeMounts:
      - name: kaniko-secret
        mountPath: /kaniko/.docker
      - name: workspace
        mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: harbor-regcred
        items:
          - key: .dockerconfigjson
            path: config.json
    - name: workspace
      hostPath:
        path: /root/kaniko/kaniko-demo

#1821 is still triggering☹