Master doesn't put to SchedulingDisabled mode

[shamil]

Hi,
Since upgrading to kops 1.4.1, the master is no longer put into Ready,SchedulingDisabled.
Initially starts as Ready,SchedulingDisabled, but after minute switches to back to Ready.

[krisnova]

Just curious - but what problems is this causing?

[chrislovecnm]

Yah I noticed this. Without it being in SchedulingDisabled mode pods are scheduled to the master. Typically you don't want your master acting as a minion.

[krisnova]

Understood. Are replication steps fairly straight forward? Is it safe to say this can be easily replicated with a fresh copy of kops 1.4.1 and a simple k8s deployment?

Attempting to replicate now, but just wondering what you have noticed - and want to make sure I am replicating the same issue.
Cheers

[justinsb]

The master is tainted, so normal pods should not schedule on it. Daemonsets don't honor taints though, but we also label it, so that you can target a daemonset only to run on the master. We don't have a label on the nodes yet, so you could target a daemonset to run only on the nodes, but we probably should.

This is the "new way" of doing things, but we're ahead of - well - everyone here. And perhaps we shouldn't have done that. Would be interested to hear what problems this causes - we should open issues in the kubernetes repo where we are finding fault with this model.

We do have two problems already:

NodePort on the master: kubernetes/kubernetes#33884

kubectl get nodes is awkward: kubernetes/kubernetes#33533

Neither is a deal-breaker AFAIK though

[shamil]

2 questions:

1. Are taints supported in k8s 1.3.x? or we need to upgrade to 1.4
2. The NodePort issue is deal-breaker no? Usually masters use smaller instances, which may affect networking as well.

[dkapanidis]

For those interested to activate again the SchedulingDisabled flag until this is fixed you can do on each master:

kubectl patch node MASTER_NAME -p "{\"spec\":{\"unschedulable\":true}}"

The issue we're facing is that the ELBs are pointing also to masters. This will disconnect the masters from the ELBs and redirect traffic only to nodes.

[chrislovecnm]

What is the status on this issue? Can we close?

[chrislovecnm]

Closing, please reopen if needs be.

[mausch]

I lost track of how k8s changed "taints"/"schedulable" flags, but I just provisioned a new cluster with kops 1.8.1 10 days ago and got an unwanted deployment on a master. Had to drain it manually.