storage: TestIntegration_SoftDelete failed

[flaky-bot]

Note: #9781 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: ec0cbb2
buildURL: Build Status, Sponge
status: failed

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (ec0cbb2), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[BrennaEpp]

=== RUN   TestIntegration_SoftDelete/grpc
    integration_test.go:4496: HardDeleteTime of soft deleted object should be equal to SoftDeleteTime+RetentionDuration, got: 2024-09-01 07:14:01.69412 +0000 UTC, expected: 2024-09-02 07:14:01.69412 +0000 UTC
    integration_test.go:4444: delete object "soft-deleteobj-20240824-25727397791346-0058" from bucket "golang-grpc-test-20240824-25727397791346-0086": storage: object doesn't exist

Looks like the same issue as before:

This is likely due to metadata propagation delay when changing soft delete retention to 9 days instead of 8 - got time is 8 days in the future, which was the original retention policy, expected time is 9 days as was updated in L4322[this line number now likely wrong].

[tritone]

Continuing to see failures on this more recently, e.g. https://btx.cloud.google.com/invocations/1af675d2-88a2-4664-bb08-2e18dc0733fa/targets/cloud-devrel%2Fclient-libraries%2Fgo%2Fgoogle-cloud-go%2Fcontinuous%2Fearliest-version/log

@BrennaEpp should we skip this or could you investigate at some point?

[frankyn]

Looks like different failure in @tritone's comment:

I'm working on this error in: #11226

Thanks for linking me @BrennaEpp