INCIDENT: Prow currently down

[Gregory-Pereira]

All prow jobs are currently failing with a 503 internal server error. It pipes a hexdump to the logs (see logs below).
prow.log

It just spits out a fat hex dump... More info to come.

/assign
/cc @harshad16

[harshad16]

This would happen now with the switch of obc to smaug
there is a delay in response now.

time aws s3 --endpoint https://s3-openshift-storage.apps.smaug.na.operate-first.cloud --profile smaug-prow ls s3://ci-prow/

An error occurred (504) when calling the ListObjectsV2 operation (reached max retries: 4): Gateway Timeout
aws s3 --endpoint  --profile smaug-prow ls s3://ci-prow/  0.42s user 0.04s system 0% cpu 2:40.74 total

This would cause issue in PR checks and logs find.

[Gregory-Pereira]

Another instance of this happening despite the OBC being moved back to smaug: https://prow.operate-first.cloud/view/s3/ci-prow/prow-logs/pr-logs/pull/operate-first_apps/2385/kubeval-validation/1567159471027785728

corresponding pr: #2385

[harshad16]

The prow jobs init container are having hard time to upload the data to storage.
init-upload is init container step in prow jobs and it timeouts before it could upload the data, to start the CI test that the prow jobs have to run.

From the logs,
It is observed that data is queued for upload, however the container timeouts due to delay in response.

{"component":"initupload","dest":"pr-logs/pull/operate-first_apps/2399/kubeval-validation/1569375544263315456/clone-records.json","file":"prow/pod-utils/gcs/upload.go:86","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2022-09-12T17:21:16Z"}
{"component":"initupload","dest":"pr-logs/pull/operate-first_apps/2399/kubeval-validation/1569375544263315456/started.json","file":"prow/pod-utils/gcs/upload.go:86","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2022-09-12T17:21:16Z"}
{"component":"initupload","dest":"pr-logs/pull/operate-first_apps/2399/kubeval-validation/1569375544263315456/clone-log.txt","file":"prow/pod-utils/gcs/upload.go:86","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload","level":"info","msg":"Queued for upload","severity":"info","time":"2022-09-12T17:21:16Z"}
{"component":"initupload","dest":"pr-logs/pull/operate-first_apps/2399/kubeval-validation/1569375544263315456/started.json","file":"prow/pod-utils/gcs/upload.go:114","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2022-09-12T17:21:16Z"}
{"component":"initupload","dest":"pr-logs/pull/operate-first_apps/2399/kubeval-validation/1569375544263315456/clone-log.txt","file":"prow/pod-utils/gcs/upload.go:114","func":"k8s.io/test-infra/prow/pod-utils/gcs.upload.func1","level":"info","msg":"Finished upload","severity":"info","time":"2022-09-12T17:25:20Z"}

Only 2/3 data is completed in upload.

464e54bb-32bf-11ed-a4d6-0a580a80025f-initupload.log

[Gregory-Pereira]

Yesterday I tested connectivity to the bucket and found no issues. I tested listing, uploading and downloading:

$ time aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow ls s3://prow
aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow l  2.16s user 0.89s system 38% cpu 7.872 total
$ time aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow cp ./testing.yaml s3://prow/
upload: ./testing.yaml to s3://prow/testing.yaml
aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow c  0.68s user 0.18s system 14% cpu 5.823 total
$ time aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow cp  s3://prow/testing.yaml ./
download: s3://prow/testing.yaml to ./testing.yaml
aws s3 --endpoint http://s3-eu-central-1.ionoscloud.com/ --profile aws-prow c  0.75s user 0.15s system 13% cpu 6.684 total

[durandom]

is everything in prow reconfigured to the new ionos storage? maybe some controllers need to be restarted to pick up the new config?

[harshad16]

Each job attaches to the credentials via secrets
These secrets seems to be correct https://console-openshift-console.apps.smaug.na.operate-first.cloud/k8s/ns/opf-ci-prow/secrets/s3-credentials

[Gregory-Pereira]

I read through the docs on IONOs storage, but I couldnt find information on configuring two of the five properties available in that secret, mainly wether to "s3_force_path_style" and wether the connection was secure or inseccure. Maybe that could be related?

[codificat]

We surely want a secure connection.

A question I have (disclaimer: I don't know/have context about ionos usage, so this might be off-topic): the endpoint is eu-central-1 (i.e. in Europe), while the cluster is in North America (right?). Can we not use a closer location?

It probably does not cause fatal issues, though. But it might be better to use a "local" endpoint

[Gregory-Pereira]

Well we currently have storage issues on smaug (slow obc connection to opf-ci-prow timeout producing frequent flakes), and infra as we are still trying to hook it up with storage from the NESE folks. It is a bucket, so I want to try to make it work first rather than provision a whole new one, remake all the storage routing changes and do this whole debugging jig again.

[durandom]

Unfortunately IONOS doesn't have a US S3 location

[codificat]

Prow jobs seem to be running successfully at the moment:

Success rate over time: 3h0s: 94%, 12h0s: 95%, 48h0s: 95%

[durandom]

@Gregory-Pereira can we close this incident?