Added tracking for deleted namespace status check in restore flow

[sangitaray2021]

Thank you for contributing to Velero!

Please add a summary of your change

Does your change fix a particular issue?

Fixes #8234

Please indicate you've done the following:

• Accepted the DCO. Commits without the DCO will delay acceptance.
• Created a changelog file (make new-changelog) or comment /kind changelog-not-required on this PR.
• Updated the corresponding documentation in site/content/docs/main.

[anshulahuja98]

For - #8234

[codecov]

Codecov Report

Attention: Patch coverage is 89.79592% with 5 lines in your changes missing coverage. Please review.

Project coverage is 58.98%. Comparing base (3f9c2dc) to head (6a5e8c2).
Report is 102 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/util/kube/resource_deletionstatus_tracker.go	76.19%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8233      +/-   ##
==========================================
- Coverage   59.15%   58.98%   -0.18%     
==========================================
  Files         367      368       +1     
  Lines       30777    38929    +8152     
==========================================
+ Hits        18206    22962    +4756     
- Misses      11113    14505    +3392     
- Partials     1458     1462       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kaovilai]

Please add changelog and run linter.

[reasonerjt]

Given we've reached FC and this may impact the restore flow, let's hold this one until the branch is cut for v1.15

[anshulahuja98]

@kaovilai can you help review the PR?

[kaovilai]

IIUC from reviewing, this change reduce potential duplicate await on if namespace exists (wait for 10 min polling) to just one instance per restore. Is that correct?

It's not meant to add polling until deletion is complete to resume some other process?

[sangitaray2021]

IIUC from reviewing, this change reduce potential duplicate await on if namespace exists (wait for 10 min polling) to just one instance per restore. Is that correct?

It's not meant to add polling until deletion is complete to resume some other process?

yes thats correct. We are reducing duplicate wait by caching the namespace which is under deletion.

[kaovilai]

lgtm

[anshulahuja98]

@reasonerjt can you review this PR now given 1.15 is cut?

[reasonerjt]

@anshulahuja98 I don't quite understand the background of the problem. I'm asking Xun @blackpiglet to take a look as he used to have discussion around this topic.

How often do you see a namespace being deleted when velero is trying to restore an object in this namespace?

[blackpiglet]

Per my understanding, this PR tracks the namespace hanging in the terminating state.
If the namespace already hits a timeout during the restore, it is added to the tracked, and a flag is marked as true.
If there is a second time meeting the same timeout issue for the namespace, the restore is marked as failed.

IMO, revoking the ongoing restore is the ideal solution for this issue, but that obviously needs more effort, so I think this is also an acceptable solution for now.

[anshulahuja98]

@reasonerjt

we have seen instances of this happening for our customers.
The issue is that if say you have 100 resources in 1 namespace to be restored, in current behaviour the code will take 100*10 minutes to wait which is not ideal in any way.
With this fix we will only wait 10 minutes per namespace.

[anshulahuja98]

@blackpiglet can you signoff if the PR looks good?