ASO does not retry after initial RBAC failure for storage accounts

[nraooptum]

When attempting to create a storage account with customer managed key encryption, if the user-assigned managed identity of the storage account does not have RBAC to the Key Vault, the operator/Azure correctly errors out for that resource saying that there is a key vault authentication failure. However, at some point the RBAC for the storage account's UAMI (user assigned managed identity) does get reconciled, but the storage account resource continues to stay stuck in this error state. The only way to recover from this is to completely delete the storage account resource and then re-create it (note: kubectl replace does not work -- the resource has to be deleted completely), which correctly deploys the storage account as it no longer gets a key vault authentication failure.

If it helps, the relevant section of my config for the storage account looks like this (it's written in typescript with cdk8s):

[matthchr]

Can you share the error message from the ASO logs? It seems like we probably ought to be retrying on that (maybe for all resources, or maybe just for the storage accounts API, it will depend on the specific message).

You also said this:

The only way to recover from this is to completely delete the storage account resource and then re-create it (note: kubectl replace does not work -- the resource has to be deleted completely), which correctly deploys the storage account as it no longer gets a key vault authentication failure.

I don't think that this is necessarily true. You'll need to trigger a fresh reconcile on the resource. Delete + recreate can do it, but you also could do something like edit a tag, or add and remove the skip-reconcile annotation.
Obviously none of those are ideal, so if you can share the error message you received here we'll update things so that we correctly retry on this.

[nraooptum]

Going to close this -- we ended up using the ArgoCD "waves" feature to ensure RBAC prior to creating the storage account.