The Clean Up GCP Resources job is flaky

[github-actions]

The Clean Up GCP Resources is failing over 80% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=1&var-Workflow=Clean%20Up%20GCP%20Resources

[Abacn]

there are more "Tried and failed to delete bq_read_all_" and other datasets. Likely related to recent python test workflow failure, which used a different service account to run tests

[github-actions]

Reopening since the workflow is still flaky

[Abacn]

due to 2.58.1 validation Python PostCommits run again without #31955 and leaked dataset

[github-actions]

Reopening since the workflow is still flaky

[Abacn]

cleaned up stale instances

[github-actions]

Reopening since the workflow is still flaky

[github-actions]

Reopening since the workflow is still flaky

[damccorm]

Cleaned up some stale resources, got a green run

[github-actions]

Reopening since the workflow is still flaky

[Abacn]

It has been a recurring issue: #33435

[Amar3tto]

Job has been stable for 2 weeks. Closing as resolved.

[Abacn]

Amar3tto@ Thanks for keeping an eye for this. This is mitigated as we skipped clean up k8s workload that has non-empty finalizer for now (#33545). However we still need to fix the clean up script so it works for such workflow (otherwise stale workload will still accumulate as time goes)

Re-opened #33435 for tracking this follow up.