Make the failurePolicy of the mutating admission controller configurable

[tkent]

Is your feature request related to a problem? Please describe.

The helm chart does not allow the failurePolicy for the mutating admission controller to be configurable. Since pods can be created or updated before the Azure Workload Identity webhook controllers have started up, they can be correctly annotated but fail to have the credential information injected into their definition.

Describe the solution you'd like

Allow the user to specify what type of failurePolicy they'd like to have in their cluster.

Describe alternatives you've considered

n/a

Additional context

This comes in scenarios where many nodes are being swapped - especially in small clusters.

[sozercan]

@tkent Gatekeeper has some considerations documented here https://open-policy-agent.github.io/gatekeeper/website/docs/failing-closed/#considerations, this is especially important since you are swapping nodes. Fail closed can cause circular dependencies in your clusters.

[tkent]

Thanks @aramase!

@sozercan - I appreciate the warning. Hopefully, in a future version of k8s, we'll be able to define scope rules that exclude the pods providing the webhook to avoid the chicken-and-egg problem (kubernetes#92157).