Automatically create ASG tag from nodegroup's labels and taints

[nikimanoledaki]

Background

The Autoscaling group (ASG) uses tags to detect and scale a nodegroup. Such tags are required for the ASG to scale a nodegroup from 0, for example if it was created with desiredCapacity: 0.

More context:

• scaling managed nodegroups to 0 #3793
• [EKS] [request]: Managed Nodes scale to 0 aws/containers-roadmap#724 (comment)
• Cluster Autoscaler Addon: Support scaling up from 0 #1481
• Support adding correct Cluster Autoscaler tags to a nodegroup without granting self-scaling permissions #2263

Problem

Currently, as per the eksctl docs, there is a bit of repetition happening in the config file when we define labels/taints and need them to be added as tags:

nodeGroups:
  - name: ng1-public
    ...
    labels:
      my-cool-label: pizza
    taints:
      feaster: "true:NoSchedule"
    tags:
      k8s.io/cluster-autoscaler/node-template/label/my-cool-label: pizza
      k8s.io/cluster-autoscaler/node-template/taint/feaster: "true:NoSchedule"

Proposed Solution

As suggested by @sammort in #3793 (comment), it would be great if the ASG tags were created automatically when labels and/or taints have been defined.

There is a limit of maximum of 50 tags for an ASG, but there is a proposal to change this for the Kubernetes cluster autoscaler. For the time being, from our side, we might want to validate that the number of labels, taints, and tags in a config are not greater than 50. Not sure if there is an error from EKS if the count is greater than 50 so this would be good to investigate too.

Another thing to keep in mind is tag duplicates.

[TBBle]

I think #1481 was looking for the same thing, but closed-out due to inactivity. Also some overlap with #2263, which proposed a NodeGroup config option, e.g. tagForAutoScaler, to add the core CA tags, and which would make sense to control this tag-copying too.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[AndrewFarley]

So, there just is no way I can figure out to scale from 0 reliably without someone implementing this feature afaik? I can't figure out all the various tag "hints" I need to provide to trick CA into scaling up from 0 reliably, especially when using nodeSelectors. Can we prioritize this?

[TBBle]

No, scale-from-zero works fine for non-managed node-groups. This ticket was just requesting that eksctl automate a bit of the currently-needed repitition, i.e. in the "problem" example, the tags values could be auto-generated based on the labels and taints values, but right now you need to write them yourself in the config.

[AndrewFarley]

@TBBle I've tried the latest releases of Cluster Autoscaler and it does not afaict support scaling from 0 without tags, but ONLY on non-managed node groups. On fully managed node groups no amount of tags helps it to work. I did figure out the tags necessary to make this work after many hours of fiddling and brute force, but EKSCTL needs to add these tags to make it work properly.

[TBBle]

I meant that it works with tags, you need to add them yourself to your eksctl config, but once you do that, it works (or did last time I deployed it, at least).

Managed node groups scale-from-zero is a different problem, that depends on some Cluster Autoscaler work by AWS (or a volunteer, I guess), see the tracking ticket and the accepted design proposal in Cluster Autoscaler that is yet to be implemented.

[Himangini]

We want to do

• automatically generate ASG tags from node labels/taints
• make sure we don't exceed the 50 tag limit and fail creation if we do so. We need to investigate if this limit may be extended. If its not then we stick to the 50 tag limit
• add a option in config file to avoid/ disable ASG tag generation for users to skip this option