Replicate webhook pods for resiliency

[imjasonh]

Currently we run one replica of the webhook deployment, which makes it a single point of failure in the event of a node outage. The job doesn't do anything that requires running as a single replica, so we should investigate running multiple replicas.

We can crib from Knative Serving, which configures:

• a HorizontalPodAutoscaler (HPA) to run 1-5 replicas based on CPU usage
• a PodDisruptionBudget to ensure that at least 80% of replicas are available
• pod anti-affinity to avoid having one node run all replicas

/kind feature

[afrittoli]

Thanks for this.
I think the pod anti-affinity will also mean that we will have one pod in case of a single-node cluster, which would be perfect as I'd like to preserve the smallest possible footprint for dev / test scenarios.

[raballew]

Awesome idea! This is not only true for a node outage but also heavy load scenarios for the event listeners. In one of our production environments we faced situations, where the event listener got unresponsive because no auto scaling was in place and too many requests were send to it.