Tests need support for simulated errors

[theunrepentantgeek]

We should cover various corner cases being able to inject certain failures. Some of this should be achievable using our existing envtest + recording framework and some may need some additional framework updates.

Ideas:

• DeploymentAlreadyExists when we attempt to create it?
• InternalServerError for a period of time from a service.
• "Create" a resource that the operator doesn't know about but which already exists.
• Deployment fails after some polling due to 5xx error.
• Throttling from a service.
• Simulated errors saving changes to spec or status (kube-apiserver instability)
• Simulated loss of entire status

Originally logged as #1726 by @matthchr

[matthchr]

This is still true and something we should investigate further

[matthchr]

No status change from above, although a lot of this can be covered with our existing unit tests so the need isn't urgent.

[theunrepentantgeek]

Still something we want to do.

[matthchr]

Still something we want to do, I have some ideas about how we could do this related to the "manually pump the reconciler" tests that I saw while helping with Karpenter.

[matthchr]

No change from above

[theunrepentantgeek]

Would be nice to have.