[AKS] Autoscale agentpool based on node-level CPU usage metrics

[nemobis]

Which component are you using?:

Cluster autoscaler

Is your feature request designed to solve a problem? If so describe the problem this feature should solve.:

While I can scale the number of pods for my workloads based on pod-level CPU usage, this isn't always enough. In some circumstances, even if all nodes are at 100 % CPU usage and autoscaling is enabled, no new nodes will be scheduled because the autoscaler only looks at CPU requests.

Describe the solution you'd like.:

I'd like to be able to configure limits in CPU usage for nodes in an agentpool, so that new nodes are added when the node-level CPU usage is above the limit, and new pods can be scheduled on non-breaching nodes only.

Describe any alternative solutions you've considered.:

The kubernetes descheduler could be used to remove pods from overloaded nodes (where the real node-level CPU usage as reported by the kernel to node-exporter is above, say, 95 %). To prevent the pod from being scheduled on the same node again, something else should take care of disabling scheduling on the overloaded node (the RemovePodsViolatingNodeTaints option would also be relevant). Once all nodes become unschedulable, a newly scheduled pod would trigger a request for a new node.

This is possible but sounds a bit roundabout, and the descheduler seems overkill.

Additional context.:

Cluster autoscaling, at least on AKS, only consider memory and CPU requests. When a node is under memory pressure, scheduling is disabled, pods get OOM'ed and eventually the rescheduled pods will cause a new node to be added. However, for CPU pressure this doesn't really happen, and pods may remain starved for CPU indefinitely.

Custom controllers are mentioned as one possibility. I don't want to build a full-blown operator for my assorted loads, when the problem is generic enough.

[nemobis]

I've filed this here as a starting point because I didn't know how to determine whether it's out of scope for the kubernetes autoscaler. I see there are some alternative node autoscalers like https://karpenter.sh with different approaches.

[nemobis]

#4016 seems somewhat related, though it would need at least something like #5153 to get real node-level metrics.

[nemobis]

I see in the FAQ that

CPU-based (or any metric-based) cluster/node group autoscalers, like GCE Instance Group Autoscaler, are NOT compatible with CA. They are also not particularly suited to use with Kubernetes in general.

https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#is-cluster-autoscaler-compatible-with-cpu-usage-based-node-autoscalers

So I suspect this feature request will be declined. The reasons seem to apply mostly to nodepool downscaling, not to upscaling. The sentence

Cluster Autoscaler makes sure that all pods in the cluster have a place to run, no matter if there is any CPU load or not.

make no sense if you're scheduling a new pod on a node which has already run out of CPU cycles.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[adrianmoisey]

/area cluster-autoscaler

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.