AzureMachinePoolMachine remove finalizer and avoid recreation if VMSS is deleting

[mweibel]

What type of PR is this?
/kind bug

What this PR does / why we need it:
This PR fixes the issues related to deleting MachinePools with VMSS still existing. Copying from the linked issue:

1. MachinePool delete deletes the VMSS
2. Finalizer on AzureMachinePoolMachine gets removed because AzureMachinePool's deletionTimestamp is non zero
3. Machine gets deleted because dependant AMPM was deleted due to removing the finalizer
4. At this point however, the VMSS VM still exists
5. Next reconcile of the AzureMachinePool recreates the AzureMachinePoolMachine again
6. Now there is no Machine anymore for that AzureMachinePoolMachine and it can't get reconciled/deleted anymore

This PR fixes point 5 by avoiding to recreate the VMSS VM when it's already in Deleting/Deleted state. Additionally it fixes Point 6 (more as a safeguard) by removing a finalizer on an AMPM early on when the AzureMachinePool is deleting and the Machine does not exist.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #4955

Special notes for your reviewer:
possibly @Jont828 could review this.

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

fixes a few edge cases related to deleting a MachinePool

[k8s-ci-robot]

Hi @mweibel. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

Attention: Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 62.23%. Comparing base (e699f04) to head (f4153d3).
Report is 79 commits behind head on main.

Files	Patch %	Lines
.../controllers/azuremachinepoolmachine_controller.go	85.71%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4959   +/-   ##
=======================================
  Coverage   62.22%   62.23%           
=======================================
  Files         201      201           
  Lines       16886    16894    +8     
=======================================
+ Hits        10508    10514    +6     
- Misses       5585     5588    +3     
+ Partials      793      792    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mboersma]

/ok-to-test

[mweibel]

thanks @mboersma. I believe the failed test is due to #4939 being not merged yet.

[mweibel]

/retest

[mweibel]

nvm latest comment - could anyone please review this change? Tests pass :)

[mboersma]

/lgtm
/assign @nojnhuh

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 89b209ef127313e094dbb69c0c6adffe03165e86

[nojnhuh]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nojnhuh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nojnhuh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mweibel]

/retest

[mweibel]

feels like those test failures are not related to my changes, but maybe I'm wrong.

[k8s-ci-robot]

@mweibel: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-e2e-with-wi-optional	f4153d3	link	false	/test pull-cluster-api-provider-azure-e2e-with-wi-optional

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.