[Feature request] Allow resource deployments into specific node groups

[AbdelrhmanHamouda]

Is your feature request related to a problem? Please describe.
Enable the operator to support deploying cluster resources into specific node groups

Describe the solution you'd like
Through the Custom resource, support a filed that would dictate which Kubernetes node group to be used.

[AbdelrhmanHamouda]

Initial thoughts are to use nodeAffinity instead of nodeSelector. Also to use requiredDuringSchedulingIgnoredDuringExecution instead of preferredDuringSchedulingIgnoredDuringExecution.

Currently investigating the definition and placement of the interface for this feature.

Tasks list:

• Investigation of possible solutions
• Definition of feature interface
• Implementation (with support for multiple labels)
• Testing
• Verification with AWS EKS nodeGroups
• Documentation

[AbdelrhmanHamouda]

Feature Plan

Support Kubernetes node affinity specifically requiredDuringSchedulingIgnoredDuringExecution.

Configuration & interface

The feature will have two methods of configuration & interface allowing for more refined control.

Method 1

CRD & CRs

• **node Affinity **: Optional section to allow the declaration of nodeSelectorTerms in a way that closely matches Kubernetes "look & feel" of declaring. The operator field value will be a direct match to everything that Kubernetes support today (i.e. In, NotIn, Exists, DoesNotExist, Gt and Lt). Reference docs.
  example:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
           matchExpressions:
               - key: disktype
              operator: In
              values:
                 - ssd  

• **Taints and Tolerations **: An Optional section to allow adding Taint tolerations to generated resources. The support will be modelled to closely follow Kubernetes Taints & Tolerations docs

Method 2

HELM

• Default affinity key/value pairs, sensible defaults will be provided by the team / individual owning the operator in the cluster. if unset / unused, the field will default to null meaning that the controller will not inject node affinity labels by default. This will follow a similar structure to the one defined in the CRD.
• Default Taint tolerations: A list of default tolerations to be added to generated resources
• Affinity Allow list, A list of allowed affinity keys & values so when an override comes through a deployed CR, the Controller will check if it is allowed or not. default is null meaning allow all overrides.
• Allow overrides, simple switch to enable / disable overrides.

[jamessthompson]

Thank you for your work on this operator.

This feature is exactly what we need. I see you've merged a few PRs to implement it, do you know when you plan to release it? If a release isn't planned soon, is it in a usable state if we compile it ourselves?

[AbdelrhmanHamouda]

Hello @jamessthompson,
Glad to see that you are utilising the project. I would be interested to hear about how it is being utilised and your experience with it so far if you are willing to share.

The feature will be released in few days at the most along with related documentation. Feel free to check the linked PR and voice your thoughts if you wish.

[jamessthompson]

This week, we determined that we need to do a large load test of our system before the end of the month. With that short timeline, we decided our first attempt would use locust, a tool I've used in the past. While the locust helm chart would work, your operator would make load tests easier to maintain in the long run, so we figured we'd start with it.

We need the functionality in this feature request to configure this specific load test to provision pods on nodes in public subnets.

[AbdelrhmanHamouda]

most of development has concluded.
Pending:

• HELM control
• AWS nodeGroups verification

[AbdelrhmanHamouda]

HELM changes have been merged and feature has been released.

Usage documentation can be found under the "Advanced Topics" section in the documentation.