Readiness concept for nodes

[hjacobs]

Upscaling or node replacement poses the risk of race conditions between application pods and required system pods (DaemonSet). We have not yet figured out a good way of postponing application scheduling until the node is fully ready. The kubelet’s Ready condition is not enough as it does not ensure that all system pods such as kube-proxy and kube2iam are running. One idea is using taints during node initialization to prevent application pods to be scheduled until the node is fully ready.

[hjacobs]

One approach would be using https://github.com/mikkeloscar/kube-node-ready-controller and wait for the taint node.alpha.kubernetes.io/notReady-workload to disappear.

/cc @mikkeloscar