Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flake] Error preparing docker image for GCB #6327

Closed
coryan opened this issue Apr 22, 2021 · 5 comments · Fixed by #6634
Closed

[Flake] Error preparing docker image for GCB #6327

coryan opened this issue Apr 22, 2021 · 5 comments · Fixed by #6634
Labels
cpp: flake A test with false positives (failures that are not interesting) type: cleanup An internal cleanup or hygiene concern.

Comments

@coryan
Copy link
Contributor

coryan commented Apr 22, 2021

Maybe a race condition? Or could be a bug in Koniko?

https://pantheon.corp.google.com/cloud-build/builds/a3f20e65-2048-4a59-9a97-f81f608b344a?project=cloud-cpp-testing-resources
@coryan coryan added the type: cleanup An internal cleanup or hygiene concern. label Apr 22, 2021
@devjgm
Copy link
Contributor

devjgm commented May 24, 2021 

error pushing image: failed to push to destination gcr.io/cloud-cpp-testing-resources/cloudbuild/fedora:a3f20e65-2048-4a59-9a97-f81f608b344a: PATCH https://gcr.io/v2/cloud-cpp-testing-resources/cloudbuild/fedora/blobs/uploads/AJAD5v0Kmaa97Q7eSqRBWNCBtHaqgtgUTHrARVmCjiX7qi4ddTPeXG0WbpDET60B-1S4Bj3pqCLax2oebsCuMVM: UNKNOWN: Unable to determine the upload's size.

@coryan
Copy link
Contributor Author

coryan commented May 25, 2021

Maybe this can help:

https://github.com/GoogleContainerTools/kaniko/blob/master/README.md#--push-retry

devjgm added a commit to devjgm/google-cloud-cpp that referenced this issue May 25, 2021
@devjgm
ci(gcb): kaniko retries pushes 3x
235164d 
This change might fix googleapis#6327
@devjgm devjgm mentioned this issue May 25, 2021
Closed
@devjgm
Copy link
Contributor

devjgm commented May 25, 2021

Interesting. The kaniko image we're using is gcr.io/kaniko-project/executor:edge, which is from last year (May 7, 2020). There are many version updates since then.

Do you know why we're using the :edge tag? I'm thinking that maybe updating to a newer version of kaniko may fix the issue (and maybe adding that --push-retry flag may help too)

@devjgm
Copy link
Contributor

devjgm commented May 25, 2021

fwiw, it turns out that newer version of a Kaniko seem to SEGV, for a reason I don't understand but I filed a bug about a couple months ago: GoogleContainerTools/kaniko#1604

@coryan
Copy link
Contributor Author

coryan commented May 25, 2021

My recollection is that I tried to use :latest that had some bugs with (maybe) multi-stage Dockerfiles, I switched to :edge and never changed the tag. This is all very depressing.

@devjgm devjgm mentioned this issue May 25, 2021
Merged
devjgm added a commit that referenced this issue May 26, 2021
@devjgm
ci(gcb): update kaniko version and retry pushes (#6634)
0264c28 
This change likely fixes #6327. 

As of today, Kaniko `v1.6.0` is the newest released version. Using that version directly results in the crash described in GoogleContainerTools/kaniko#1604. However, using the `v1.6.0-debug` image works, I _think_ because of a different version of the `docker-credentials-gcr` helper as I described in GoogleContainerTools/kaniko#1604 (comment)

Updating Kaniko may also help #6336
@coryan coryan added the cpp: flake A test with false positives (failures that are not interesting) label Jan 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cpp: flake A test with false positives (failures that are not interesting) type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Milestone
No milestone
2 participants
@coryan @devjgm