Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

INTERNAL_ERROR building image with Kaniko #6336

Closed
coryan opened this issue Apr 23, 2021 · 16 comments
Closed

INTERNAL_ERROR building image with Kaniko #6336

coryan opened this issue Apr 23, 2021 · 16 comments
Labels
cpp: flake A test with false positives (failures that are not interesting) type: cleanup An internal cleanup or hygiene concern.

Comments

@coryan
Copy link
Contributor

coryan commented Apr 23, 2021

Error in Kaniko step:

INFO[0047] Found cached layer, extracting to filesystem 
error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 95; INTERNAL_ERROR

Looks like 3 builds affected last night:

https://pantheon.corp.google.com/cloud-build/builds;region=global/47becb5f-2556-41d4-8b67-e0ee160a68f4;step=0?project=cloud-cpp-testing-resources
https://pantheon.corp.google.com/cloud-build/builds/af84399a-58de-4e62-a2fb-84991c1fd9d2;step=0?project=cloud-cpp-testing-resources
https://pantheon.corp.google.com/cloud-build/builds;region=global/cbb56412-49cd-4b9d-b3f3-610b65b19d9e;step=0?project=cloud-cpp-testing-resources
@coryan coryan added the type: cleanup An internal cleanup or hygiene concern. label Apr 23, 2021
@coryan
Copy link
Contributor Author

coryan commented May 3, 2021

https://pantheon.corp.google.com/cloud-build/builds/07dea1f5-b889-43c0-84e0-0e84f9fd9c9f;step=0?project=cloud-cpp-testing-resources

Step #0: INFO[0040] No files changed in this command, skipping snapshotting. 
Step #0: INFO[0040] RUN /var/tmp/ci/install-cloud-sdk.sh         
Step #0: INFO[0040] Found cached layer, extracting to filesystem 
Step #0: error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 93; INTERNAL_ERROR
Finished Step #0

@devjgm devjgm mentioned this issue May 25, 2021
Merged
devjgm added a commit that referenced this issue May 26, 2021
@devjgm
ci(gcb): update kaniko version and retry pushes (#6634)
0264c28 
This change likely fixes #6327. 

As of today, Kaniko `v1.6.0` is the newest released version. Using that version directly results in the crash described in GoogleContainerTools/kaniko#1604. However, using the `v1.6.0-debug` image works, I _think_ because of a different version of the `docker-credentials-gcr` helper as I described in GoogleContainerTools/kaniko#1604 (comment)

Updating Kaniko may also help #6336
@devbww
Copy link
Contributor

devbww commented Jul 22, 2021

https://console.cloud.google.com/cloud-build/builds;region=us-east1/42a7f779-924d-4533-bb50-e4350f41fbe7?project=cloud-cpp-testing-resources

Step #0: time="2021-07-21T21:25:21Z" level=info msg="RUN /var/tmp/ci/install-cloud-sdk.sh"
Step #0: time="2021-07-21T21:25:21Z" level=info msg="Found cached layer, extracting to filesystem"
Step #0: error building image: error building stage: failed to execute command: extracting fs from image: stream error: stream ID 3; INTERNAL_ERROR
Finished Step #0
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:v1.6.0-debug" failed: step exited with non-zero status: 1

@coryan
Copy link
Contributor Author

coryan commented Aug 20, 2021

https://console.cloud.google.com/cloud-build/builds;region=us-east1/66c3d69a-7c8e-441d-8150-1f18fa8ab38d?project=936212892354

and

https://console.cloud.google.com/cloud-build/builds;region=us-east1/02a28e28-7899-4cdd-9730-442b3b3cf98c?project=936212892354

@dbolduc
Copy link
Member

dbolduc commented Sep 1, 2021

https://pantheon.corp.google.com/cloud-build/builds;region=us-east1/69a2d3d6-8356-47a7-ab0d-f80081c1e460?project=cloud-cpp-testing-resources

@coryan
Copy link
Contributor Author

coryan commented Oct 20, 2021

kaniko:1.7.0 is out, it supports retries when extracting the filesystem:

GoogleContainerTools/kaniko#1685
https://github.com/GoogleContainerTools/kaniko/releases/tag/v1.7.0

A simple upgrade did not work. Seems like some configuration is needed.

@dbolduc
Copy link
Member

dbolduc commented Dec 21, 2021

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/7770/79662afb6d35baabcd7645103517b0b3aaa068d9/fedora-34-ubsan/log-9df4d9fb-dff0-4d1d-aef1-2eb1189cd722.txt
https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/7770/79662afb6d35baabcd7645103517b0b3aaa068d9/fedora-34-coverage/log-a200cc61-c10c-466e-8003-37d322249c27.txt

@coryan coryan added the cpp: flake A test with false positives (failures that are not interesting) label Jan 27, 2022
@coryan coryan changed the title [Flake] INTERNAL_ERROR building image with Kaniko INTERNAL_ERROR building image with Kaniko Jan 27, 2022
@dbolduc
Copy link
Member

dbolduc commented Feb 10, 2022

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/b8452f2f-03c9-4dc7-b147-525d759feed6?project=936212892354

@dbolduc
Copy link
Member

dbolduc commented Feb 16, 2022

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/8393/c2c30f3ea1fa2eff0d0b4a61febfd3ee449c6448/demo-ubuntu-focal-demo-install/log-34251808-f6b6-4b9e-849b-58c928cf1176.txt

@dbolduc
Copy link
Member

dbolduc commented Feb 16, 2022

hmmm another demo-ubuntu-focal....

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/811ece9fc33e5c5752a9f5ddc4946d8cc62e39ed/demo-ubuntu-focal-demo-install/log-a17ff190-ec48-4c51-a6e5-a2377ae9afb0.txt

@devbww
Copy link
Contributor

devbww commented Feb 18, 2022

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/60659dd4-1530-4eba-9e82-734f0e439933?project=936212892354

@coryan
Copy link
Contributor Author

coryan commented May 23, 2022

No repeats in 90d, and probably fixed by #8558

@coryan coryan closed this as completed May 23, 2022
@dbolduc
Copy link
Member

dbolduc commented Jul 12, 2022

(internal link) https://console.cloud.google.com/cloud-build/builds;region=us-east1/1d174b5a-4a85-45dd-b374-4f7a43287e94?project=936212892354

@dbolduc dbolduc reopened this Jul 12, 2022
@coryan
Copy link
Contributor Author

coryan commented Nov 9, 2022

No repeats in 90d, but it seems it happens on longer cycles. I will give it another chance.

@dbolduc
Copy link
Member

dbolduc commented Feb 23, 2023

https://storage.cloud.google.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/8290acb981944ce53e58e09db2df2e3a2e1420ec/ubuntu-focal-quickstart-cmake/log-6a7e6d8b-7802-4b67-adfa-c6e4425ce385.txt

@dbolduc
Copy link
Member

dbolduc commented Mar 21, 2023

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/17b0de962ef2735fd33882f6b556b322af4229a0/fedora-37-cmake-check-api/log-50e153ab-c3e6-404e-ab0b-36694048a286.txt

https://storage.googleapis.com/cloud-cpp-community-publiclogs/logs/google-cloud-cpp/main/17b0de962ef2735fd33882f6b556b322af4229a0/ubuntu-focal-quickstart-cmake/log-6f371eb1-86ec-4b82-ab99-35446ec2b4fe.txt

@coryan
Copy link
Contributor Author

coryan commented Jul 12, 2023

No repeats in 90d, closing for now.

@coryan coryan closed this as completed Jul 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
cpp: flake A test with false positives (failures that are not interesting) type: cleanup An internal cleanup or hygiene concern.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@coryan @devbww @dbolduc