Enhance Helm with an option to not generate a new IngressClass object

[brianehlert]

Discussed in #4179

^{Originally posted by brianehlert August 3, 2023}
One upgrade pattern for services in Kubernetes is a blue/green. We tend to focus on this pattern for backend services behind the ingress controller. However, this is a valid upgrade pattern for the ingress controller itself.

This is currently not possible using the Helm chart provided with this project.
What the Helm chart enforces is that each ingress controller deployment has a unique IngressClass.
While this is true to the IngressClass intention of being a unique management construct, it blocks the possibility of using the blue/green pattern to upgrade the ingress controller service itself.
The additional impact of this behavior is that the customer is not able to generate an IngressClass out of band with the Helm chart, such as part of some automation workflow prior to deploying the ingress controller using the Helm chart.

This is believed to be possible with manifest (not validated) however that does not meet the automation needs when a customer has expressly instrumented the Helm chart.

What I am proposing is some option of the Helm chart to:

• not generate a new ingress class when using the Helm chart to deploy ingress controller
• support adding an additional deployment of ingress controller to an existing IngressClass

AC:

• Ability to install IC without requiring/creating an ingressClass.
• Attach an existing ingressClass to an IC spec

Background:

This article describes the blue/green pattern for an ingress controller starting at Step 1
https://medium.com/codecademy-engineering/kubernetes-nginx-and-zero-downtime-in-production-2c910c6a5ed8
The core customer objective is that a secondary ingress controller deployment is created and both will exist for a short period of time. The customer is not concerned with the problem of two deployments writing back status to the same set of configuration objects. In fact, it is the ability to have one set of configuration objects that both deployments are sharing that makes this pattern compelling in the first place.

ingress-nginx Helm has a way to disable the creation of the IngressClass resource: https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/templates/controller-ingressclass.yaml#L1

### Tasks
- [ ] Document helm update