Horizontal Scaling RC to scale another controller based on number of cores and nodes

[girishkalele]

Split out Godeps commit from real changes

[davidopp]

cc/ @piosz @fgrzadkowski @jszczepkowski @mwielgus you might be interested in this (I suggest going to the commit view and just looking at the non-Godep commit)

@girishkalele can you say something about when people should use this vs. the Horizontal Pod Autoscaling feature?

[davidopp]

Oh and @girishkalele it might be useful to copy the README from the repo into this PR thread, since it's not completely obvious that that's where to look to understand what is the goal of this PR.

[girishkalele]

Horizontal Self Scaler container

This container image watches over the number of schedulable cores and nodes in the cluster and resizes the number of replicas in the required controller.

Usage of pod_nanny:
    --configmap <params>
    --rc <replication-controller>
    --rs <replica set>
    --deployment <deployment>
    --verbose
    --namespace <namespace>

Implementation Details

The code in this module is a Kubernetes Golang API client that, using the default service account credentials available to Golang clients running inside pods, it connects to the API server and polls for the number of nodes and cores in the cluster.
The scaling parameters and data points are provided via a ConfigMap to the autoscaler and it refreshes its parameters table every poll interval to be up to date with the latest desired scaling parameters.

Calculation of number of replicas

The desired number of replicas is computed by lookup up the number of cores using the step ladder function.
The step ladder function uses the datapoints from the configmap.
This may be later extended to more complex interpolation or linear/exponential scaling schemes
but it currently supports (and defaults to) to mode=step only.

Configmap controlling parameters

The ConfigMap provides the configuration parameters, allowing on-the-fly changes without rebuilding or restarting the scaler containers/pods.

Example rc file

This example-rc.yaml is an example Replication Controller where the nannies in all pods watch and resize the RC replicas.

[girishkalele]

This is the horizontal scaling version of the vertical addon resizer by @Q-Lee

Prior discussion here: kubernetes-retired/contrib#1427

[girishkalele]

The Horizontal Pod Autoscaler is a top-level Kubernetes API resource. It is a true closed loop autoscaler which monitors CPU utilization of the pods and scales the number of replicas automatically. It requires the CPU resources to be defined for all containers in the target pods and also requires heapster to be running to provide CPU utilization metrics.

This horizontal self scaler is a DYI container (because it is not a Kubernetes API resource) that provides a simple control loop that watches the cluster size and scales the target controller. The actual CPU or memory utilization of the target controller pods is not an input to the control loop, the sole inputs are number of schedulable cores and nodes in the cluster.
There is no requirement to run heapster and/or provide CPU resource limits as in HPAs.

The configmap provides the operator with the ability to tune the replica scaling explicitly.

[thockin]

why do we need this rule?

[thockin]

if we move this to its own RC, is "self-scaler" still a valid name? We can rename the repo...

why is everything in an "autoscaler" subdir, rather than the root?

[thockin]

Also, can I beg to derive your Makefile from kubernetes/build/pause/Makefile

On Thu, Aug 4, 2016 at 9:54 PM, Tim Hockin [email protected] wrote:

if we move this to its own RC, is "self-scaler" still a valid name? We can
rename the repo...

why is everything in an "autoscaler" subdir, rather than the root?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVMmxP9pRIeLo8S8F-XfwRR9pDZRFks5qcsINgaJpZM4JdKUR
.

[davidopp]

it seems that you only use the number of cores, not the number of nodes?

[girishkalele]

Yes, @thockin commented above about the same - we need two scale maps, one for cores and one for nodes, and we lookup both maps and choose the greater number. The user may choose to omit one map (and just scale only by number of cores or nodes). I am changing it to accept the two scale maps.

[piosz]

@girishkalele this functionality seems to be a perfect fit for custom metric case in HPA like number-of-supported-cores. The source of the the metric would be apiserver, so Heapster is not a requirement here.

I can see many benefits of having this feature in HPA:

• lower maintenance overhead
• well defined configuration
• cluster operation working out of the box (see Guaranteed scheduling of critical pods kubernetes/kubernetes#29023)

Any reasoning for having this as a separate project?

[mwielgus]

I strongly agree with @piosz. We should reuse the existing scaling infrastructure and have a consolidated pod scaling solution.

The only differences between this project and HPA are that it uses a slightly different metric to calculate desired replica count and runs in a separate pod (instead of being a controller) what makes it vulnerable to scheduling problems.

[piosz]

cc @wojtek-t

[girishkalele]

@piosz @mwielgus

The intent of this was to create a nanny container similar to the addon-resizer container used by fluentd for DNS horizontal scaling. HPA+Heapster was too heavy in resource utilization for a simple scaler.

This is also a nice base template for folks doing DYI scalers of their own, scaling along various metrics.

I didn't know about the ability of the HPA to scale using custom metrics - can it do this already today ?

[piosz]

The reason why addon-resizer is a separate container is that there is no Vertical Autoscaler in Kubernetes yet, but we have a production ready solution for horizontal scaling. While I'm ok with having this feature in the shape you propose as a temporary hack for 1.4, I think it should eventually become a part of HPA.

I don't think encouraging users to write their own scalers is the right approach - we should provide powerful api to scale based on custom metrics instead.

There is no custom metrics in HPA yet, but the problem you're trying to solve is a good reason to increase priority of it. Also see kubernetes/kubernetes#28628

[girishkalele]

@MrHohn This is the WIP on the cluster-proportional-autoscaler.

The skydns yaml template changes that add this to the kube-dns pod are here kubernetes/kubernetes#32019