Added NifiNodeGroupAutoscaler with basic scaling strategies

[mh013370]

Q	A
Bug fix?	no
New feature?	yes
API breaks?	no
Deprecations?	no
Related tickets
License	Apache 2.0

What's in this PR?

This PR contains an implementation of the autoscaling design voted on here: https://konpytika.slack.com/archives/C035FHN1MNG/p1650031443231799

It adds:

• A new custom resource called NifiNodeGroupAutoscaler
• Adds Nodes.Spec.Labels so that nodes in a NifiCluster can be tagged with arbitrary labels and specifically for the NifiNodeGroupAutoscaler to identify nodes to manage
• A new controller for this custom resource. Its reconciliation loop is as follows:
  • Fetch the current NifiCluster.Spec.Nodes list, filter it by the nodes within to manage by the provided NifiNodeGroupAutoscaler.Spec.NodeLabelsSelector
  • Compare the managed node list with the current NifiNodeGroupAutoscaler.Spec.Replicas setting and determine if scaling needs to happen. If the # replicas > # managed nodes, then scale up. If the # replicas < # managed nodes, then scale down. Else do nothing.
  • If a scale event happened, patch the NiFiCluster.Spec.Nodes list with the added/removed nodes & update the scale subresource status fields.
• A NifiNodeGroupAutoscaler can manage any subset of nodes in a NifiCluster up to the entire cluster. With this, you can enable highly resourced (mem/cpu) node groups for volume bursts or to just autoscale entire clusters driven by metrics.
• I significantly reduced the verbosity of the logs by default since it was difficult to track what the operator was actually doing. In general, i think this should be continued to the point where the operator only logs what actions it has taken or changes it has made in k8s.

I don't necessarily consider this PR complete, which is why its in draft status. See the additional context below. However, i have tested this on a live system and it does work.

Why?

To enable horizontal scaling of nifi clusters.

Additional context

There are a few scenarios that need to be addressed prior to merging this:

• On a scale up event, when the NifiNodeGroupAutoscaler adds nodes to the NifiCluster.Spec.Nodes list, the nifi cluster controller correctly adds a new pod to the deployment. However, when that node comes completely up and is Running in k8s, nifikop kills it and kicks off a RollingUpgrade and basically restarts the new node. This occurs here, but i'm not exactly sure what causes it. Scaling down happens "gracefully"

• It's not possible to deploy a NifiCluster with only autoscaled node groups. The NifiCluster CRD requires that you specify at least one node in the spec.nodes list. Do we want to support this? If so, we may need to adjust the cluster initialization logic in the NifiCluster controller.

I did all of my testing via ArgoCD. When the live state differs from the state in git, ArgoCD immediately reverts it so I had to craft my applications carefully to avoid ArgoCD undoing the changes that the HorizontalPodAutoscaler and NifiNodeGroupAutoscaler controllers were making.

I've tested scaling up and down and successfully configured the HorizontalPodAutoscaler to pull nifi metrics from Prometheus. Here's the autoscaler config that i used for that setup:

apiVersion: nifi.konpyutaika.com/v1alpha1
kind: NifiNodeGroupAutoscaler
metadata:
  labels:
    argocd.argoproj.io/instance: as-nf
  name: as-nf-scale-group
  namespace: nifi
spec:
  clusterRef:
    name: as-nf
    namespace: nifi
  nodeConfigGroupId: scale-group
  nodeLabelsSelector:
    matchLabels:
      scale_me: 'true'
  upscaleStrategy: simple
  downscaleStrategy: lifo

And the nifi_amount_items_queued_sum metric is added to the prometheus-adapter deployment as follows:

prometheus-adapter:
  rules:
    custom:
    - seriesQuery: 'nifi_amount_items_queued'
      resources:
        # skip specifying generic resource<->label mappings, and just
        # attach only pod and namespace resources by mapping label names to group-resources
        overrides:
          kubernetes_namespace:
            resource: "namespace"
          kubernetes_pod_name: 
            resource: "pod"
      name:
        matches: "^(.*)"
        as: "${1}_sum"
      metricsQuery: (sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>))

Checklist

• Implementation tested
• Error handling code meets the guideline
• Logging code meets the guideline
• User guide and development docs updated (if needed)
• Append changelog with changes

[mh013370]

Okay i've updated this PR with the following suggested changes:

• The operator no longer creates a HorizontalPodAutoscaler. This is expected to be created externally
• The way new nodeIds are assigned for autoscaling strategies is now centralized and should be shared by all following HorizontalUpscaleStrategy implementations
• Added optional NifiNodeGroupAutoscaler.Spec.NodeConfig
• Added NifiCluster.Status.NodesState.CreationTime to capture when a node was added to the cluster. This is used by the LIFOHorizontalDownscaleStrategy to remove the most recently added nodes on a scale-down event

I'll update the PR description to reflect this.

[mh013370]

Here are some example logs from upscale and downscale events:

Mon, May 16 2022 4:04:06 pm | {"level":"info","ts":1652713446.1393616,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Replicas changed from 3 to 1"}
Mon, May 16 2022 4:04:06 pm | {"level":"info","ts":1652713446.1590438,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Removing 2 nodes from cluster as-nf spec.nodes configuration for node group scale-group"}
Mon, May 16 2022 4:04:06 pm | {"level":"info","ts":1652713446.1590834,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Using LIFO downscale strategy for cluster as-nf node group scale-group"}
Mon, May 16 2022 4:05:53 pm | {"level":"info","ts":1652713553.9995086,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Replicas changed from 1 to 2"}
Mon, May 16 2022 4:05:54 pm | {"level":"info","ts":1652713554.0240824,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Adding 1 more nodes to cluster as-nf spec.nodes configuration for node group scale-group"}
Mon, May 16 2022 4:05:54 pm | {"level":"info","ts":1652713554.024129,"logger":"controllers.NifiNodeGroupAutoscaler","msg":"Using Simple upscale strategy for cluster as-nf node group scale-group"}

[mh013370]

I'm happy to squash all of these commits if that's desired.

[mh013370]

Just following up on this PR. I believe it's functionally complete with one exception.

When a node group is scaled up (e.g. the NifiCluster.spec.nodes list is patched), NiFiKop will correctly create the new pod. However, when the pod status eventually is ready & running, NiFiKop kills the pod and runs a graceful upscale cluster task effectively restarting the new pod.

Is there a way we can tell the operator to run an upscale operation to begin with to avoid this pod restarting?

Once this is resolved, i will remove the draft status of this PR so that we can move to merge it.

[erdrix]

Just following up on this PR. I believe it's functionally complete with one exception.

When a node group is scaled up (e.g. the NifiCluster.spec.nodes list is patched), NiFiKop will correctly create the new pod. However, when the pod status eventually is ready & running, NiFiKop kills the pod and runs a graceful upscale cluster task effectively restarting the new pod.

Is there a way we can tell the operator to run an upscale operation to begin with to avoid this pod restarting?

Once this is resolved, i will remove the draft status of this PR so that we can move to merge it.

I will merge this PR on a branch and create a PR to remove the pod restart because with the latest version on NiFi it is no longer necessary.

[mh013370]

Just following up on this PR. I believe it's functionally complete with one exception.
When a node group is scaled up (e.g. the NifiCluster.spec.nodes list is patched), NiFiKop will correctly create the new pod. However, when the pod status eventually is ready & running, NiFiKop kills the pod and runs a graceful upscale cluster task effectively restarting the new pod.
Is there a way we can tell the operator to run an upscale operation to begin with to avoid this pod restarting?
Once this is resolved, i will remove the draft status of this PR so that we can move to merge it.

I will merge this PR on a branch and create a PR to remove the pod restart because with the latest version on NiFi it is no longer necessary.

Sounds good! I can help review & test these changes on that PR.