Allow installing IC without creating a new ingress class

[haywoodsh]

Proposed changes

Add controller.ingressClass.create option to options to create a new ingress class object or use an existing ingress class during deployment

closes #4209

Test

1. Create an IngressClass manually.

   kubectl create -f - <<EOF
   apiVersion: networking.k8s.io/v1
   kind: IngressClass
   metadata:
     name: test-ingress-class
   spec:
     controller: nginx.org/ingress-controller
   EOF
   

2. Navigate to the helm chart folder.

   cd deployments/helm-chart
   

3. Deploy Ingress Controller with helm without creating a new IngressClass

   helm install nginx-ingress-test-ingress-class . --set controller.ingressClass.name=test-ingress-class --set controller.ingressClass.create=False
   NAME: nginx-ingress-test-ingress-class
   LAST DEPLOYED: Fri Sep  8 20:10:36 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   The NGINX Ingress Controller has been installed.
   

The IC is successfully installed with an existing IngressClass object.

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[codecov]

Codecov Report

Merging #4333 (4462415) into main (3b91d97) will decrease coverage by 0.09%.
Report is 1 commits behind head on main.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #4333      +/-   ##
==========================================
- Coverage   52.19%   52.11%   -0.09%     
==========================================
  Files          59       59              
  Lines       16929    16956      +27     
==========================================
  Hits         8836     8836              
- Misses       7796     7823      +27     
  Partials      297      297              

Files Changed	Coverage Δ
internal/configs/configurator.go	37.68% <0.00%> (-0.27%)	⬇️
internal/k8s/controller.go	11.93% <0.00%> (-0.08%)	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ADubhlaoich]

LGTM, docs-wise!

[vepatel]

@haywoodsh upgrading existing installation using:

• diff ingressClass and setting create: True, eg: name: nginx2 removes the old ingressClass
• same ingressClass and setting create: False removes the old ingressClass

I0914 11:43:45.163449       1 main.go:236] Kubernetes version: 1.25.13
F0914 11:43:45.167742       1 main.go:251] Error when getting IngressClass nginx: ingressclasses.networking.k8s.io "nginx" not found

Is this expected?

[vepatel]

If two IC deployments are sharing same ingress class then is it possible to switch between the two for serving the requests, but looks like that the ADDRESS field in Ingress is only updated by last deployed IC.
Not the case with different ingressclass as it gets updated after reload.
cc@brianehlert

[haywoodsh]

@haywoodsh upgrading existing installation using:

• diff ingressClass and setting create: True, eg: name: nginx2 removes the old ingressClass
• same ingressClass and setting create: False removes the old ingressClass

I0914 11:43:45.163449       1 main.go:236] Kubernetes version: 1.25.13
F0914 11:43:45.167742       1 main.go:251] Error when getting IngressClass nginx: ingressclasses.networking.k8s.io "nginx" not found

Is this expected?

Appreciate your testing @vepatel! Create: True is equivalent to deploying the IC without this parameter in current versions. Did you initially install an older IC version without the parameter or create: True, and then use helm upgrade to upgrade with create: False? I didn't expect this PR would handle such a scenario. If the prior IngressClass was installed and managed by helm, I can see Helm deleting it during upgrade when you don't include it in the new release with create: False.
I wonder if there is a way to use schema to enforce a constant value, but at the very least, I should make it clear in the documentation this is not recommended for users to switch the value if they're using helm upgrade.

[vepatel]

Thanks @haywoodsh there's a doc task linked in #4209 now 👍🏼
I installed 3.2.1 with default parameters (as 3.2.1 helm-chart doesn't have create option) and then upgraded by setting create: false in new helm-chart so helm deleted the old one.

[vepatel]

requesting some changes