design for running multiple velero backups/restores concurrently

[skriss]

No description provided.

[sseago]

With this feature, we're moving away from the idea of creating separate pods for concurrent backups (either with or without Jobs), mentioned in previous design PRs linked here, for a few reasons:

1. We should be able to use the shared informer cache's
2. that spinning pods will take an increasing amount of resources, and because they are all using their own clients, can DDOS the api server if we are not careful
3. That many other things have complicated tasks that are done in a single controller manager (see pods/kubelet for example)

[ywk253100]

In the first step, we could introduce the support for vertical expansion by making the controller-runtime worker count configurable(default value is 1).

[reasonerjt]

Per discussion we'll start the investigation in v1.13 timeframe, but the actual final design may go beyond 1.13

Potentially we can deal with concurrent backup/restore separately as restore may have more challenge in race condition.