Config update detection for runtime config is too slow

[pkosiec]

Overview

This is a follow-up of #704.

Currently, Config Watcher monitors config files, and once it detects changes, it exits the app. Our ConfigMaps and Secrets are mounted as files to the container (see the Deployment manifest).

However, there is one limitation - until Kubernetes mounts updated ConfigMaps, it may take a while:
https://kubernetes.io/docs/concepts/configuration/configmap/#mounted-configmaps-are-updated-automatically
How much? In my case, it could take a few seconds or even 30+s sometimes.

As we rely on the BotKube config reload for @BotKube edit SourceBindings command, this is something we should address.

We can think about reworking Config Watcher to purely support just ConfigMaps and Secrets via informers.

Also, there's also one more problem. When we restart the Pod too often, Kubernetes treats it as a crash loop even if it returns exit code 0 like us (see issue). That means, if people use the Source Bindings edit too much, Kubernetes applies the following policy:

After containers in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s, 40s, …), that is capped at five minutes. Once a container has executed for 10 minutes without any problems, the kubelet resets the restart backoff timer for that container.

Unfortunately this is not configurable by us. We should workaround this.

Acceptance Criteria

• Fix issue with crash loop
• Use ConfigMap informers for ConfigWatcher
• Make sure it works fast when it comes to botkube-runtime-config
• Document new changes

Reason

Config update detection is too slow and we rely on it when doing Source Bindings edit.