Merge pull request #40108 from MrHohn/addon-ensure-exist

Automatic merge from submit-queue

Supports 'ensure exist' class addon in Addon-manager

Fixes #39561, fixes #37047 and fixes #36411. Depends on #40057.

This PR splits cluster addons into two categories:
- Reconcile: Addons that need to be reconciled (`kube-dns` for instance).
- EnsureExists: Addons that need to be exist but changeable (`default-storage-class`).

The behavior for the 'EnsureExists' class addon would be:
- Create it if not exist.
- Users could do any modification they want, addon-manager will not reconcile it.
- If it is deleted, addon-manager will recreate it with the given template.
- It will not be updated/clobbered during upgrade.

As Brian pointed out in [#37048/comment](kubernetes/kubernetes#37048 (comment)), this may not be the best solution for addon-manager. Though #39561 needs to be fixed in 1.6 and we might not have enough bandwidth to do a big surgery.

@mikedanese @thockin 

cc @kubernetes/sig-cluster-lifecycle-misc 

---

Tasks for this PR:
- [x] Supports 'ensure exist' class addon and switch to use new labels in addon-manager.
- [x] Updates READMEs regarding the new behavior of addon-manager.
- [x] Updated `test/e2e/addon_update.go` to match the new behavior.
- [x] Go through all current addons and apply the new labels on them regarding what they need.
- [x] Bump addon-manager and update its template files.

cluster/addons/storage-class/openstack/default.yaml

@@ -6,4 +6,5 @@ metadata:
     storageclass.beta.kubernetes.io/is-default-class: "true"
   labels:
     kubernetes.io/cluster-service: "true"
+    addonmanager.kubernetes.io/mode: EnsureExists
 provisioner: kubernetes.io/cinder