Trigger automated actions based on config and fix event issues

[pkosiec]

Description

Changes proposed in this pull request:

• Load automation config and support source bindings for action commands
• Add E2E tests for Actions
• Validate Action bindings

While testing I also found, debugged and fixed the following issues:

• Fix listening to sources for communication platforms which are not configured
• Fix recommendation-related event skipping: support source-wide namespaces
• Fix not incoming cluster-wide events

Testing

Check out this PR: gh pr checkout 848

Ensure that you run Botkube with the following actions and sources local config:

# Format: actions.{alias}
actions:
  'show-created-resource':
    # -- If true, enables the action.
    enabled: false
    # -- Action display name posted in the channels bound to the same source bindings.
    displayName: "Display created resource"
    # -- A text value denoting the command run by this action, may contain even based templated values.
    # The executor is inferred directly from the command, e.g. here we require a kubectl executor
    command: "kubectl describe {{ .Event.TypeMeta.Kind | lower }}{{ if .Event.Namespace }} -n {{ .Event.Namespace }}{{ end }} {{ .Event.Name }}"

    # -- Bindings for a given action.
    bindings:
      # -- Sources of events that trigger a given action.
      sources:
        - k8s-create-events
      # -- Executors configuration for a given automation.
      executors:
        - kubectl-read-only
  'show-logs-on-error':
    # -- If true, enables the action.
    enabled: false

    # -- Action display name posted in the channels bound to the same source bindings.
    displayName: "Show logs on error"
    # -- A text value denoting the command run by this action, may contain even based templated values.
    # The executor is inferred directly from the command, e.g. here we require a kubectl executor
    command: "kubectl logs {{ .Event.TypeMeta.Kind | lower }}/{{ .Event.Name }} -n {{ .Event.Namespace }}"

    # -- Bindings for a given action.
    bindings:
      # -- Sources of events that trigger a given action.
      sources:
        - k8s-err-with-logs-events
      # -- Executors configuration for a given automation.
      executors:
        - kubectl-read-only

sources:
  'k8s-recommendation-events':
    displayName: "Kubernetes Recommendations"
    # Describes Kubernetes source configuration.
    kubernetes:
      # Describes configuration for various recommendation insights.
      recommendations:
        # Recommendations for Pod Kubernetes resource.
        pod:
          # If true, notifies about Pod containers that use `latest` tag for images.
          noLatestImageTag: true
          # If true, notifies about Pod resources created without labels.
          labelsSet: true
        # Recommendations for Ingress Kubernetes resource.
        ingress:
          # If true, notifies about Ingress resources with invalid backend service reference.
          backendServiceValid: true
          # If true, notifies about Ingress resources with invalid TLS secret reference.
          tlsSecretValid: true

  'k8s-all-events':
    displayName: "Kubernetes Info"
    # Describes Kubernetes source configuration.
    kubernetes:
      # Describes namespaces for every Kubernetes resources you want to watch or exclude.
      # These namespaces are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own namespaces object.
      namespaces: &k8s-events-namespaces
        # Include contains a list of allowed Namespaces.
        # It can also contain a regex expressions:
        #  `- ".*"` - to specify all Namespaces.
        include:
          - ".*"
        # Exclude contains a list of Namespaces to be ignored even if allowed by Include.
        # It can also contain a regex expressions:
        #  `- "test-.*"` - to specif all Namespaces with `test-` prefix.
        # exclude: []

      # Describes events for every Kubernetes resources you want to watch or exclude.
      # These events are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own events object.
      events:
        - create
        - delete
        - error

      # Describes the Kubernetes resources you want to watch.
      resources:
        - name: v1/pods             # Name of the resource. Resource name must be in group/version/resource (G/V/R) format
                                    # resource name should be plural (e.g apps/v1/deployments, v1/pods)

        #  namespaces:             # Overrides 'source'.kubernetes.namespaces
        #    include:
        #      - ".*"
        #    exclude: []
        - name: v1/services
        - name: networking.k8s.io/v1/ingresses
        - name: v1/nodes
        - name: v1/namespaces
        - name: v1/persistentvolumes
        - name: v1/persistentvolumeclaims
        - name: v1/configmaps
        - name: rbac.authorization.k8s.io/v1/roles
        - name: rbac.authorization.k8s.io/v1/rolebindings
        - name: rbac.authorization.k8s.io/v1/clusterrolebindings
        - name: rbac.authorization.k8s.io/v1/clusterroles
        - name: apps/v1/daemonsets
          events: # Overrides 'source'.kubernetes.events
            - create
            - update
            - delete
            - error
          updateSetting:
            includeDiff: true
            fields:
              - spec.template.spec.containers[*].image
              - status.numberReady
        - name: batch/v1/jobs
          events: # Overrides 'source'.kubernetes.events
            - create
            - update
            - delete
            - error
          updateSetting:
            includeDiff: true
            fields:
              - spec.template.spec.containers[*].image
              - status.conditions[*].type
        - name: apps/v1/deployments
          events: # Overrides 'source'.kubernetes.events
            - create
            - update
            - delete
            - error
          updateSetting:
            includeDiff: true
            fields:
              - spec.template.spec.containers[*].image
              - status.availableReplicas
        - name: apps/v1/statefulsets
          events: # Overrides 'source'.kubernetes.events
            - create
            - update
            - delete
            - error
          updateSetting:
            includeDiff: true
            fields:
              - spec.template.spec.containers[*].image
              - status.readyReplicas
       ## Custom resource example
       # - name: velero.io/v1/backups
       #   namespaces:
       #     include:
       #       - ".*"
       #     exclude:
       #       -
       #   events:
       #     - create
       #     - update
       #     - delete
       #     - error
       #   updateSetting:
       #     includeDiff: true
       #     fields:
       #       - status.phase

  'k8s-err-events':
    displayName: "Kubernetes Errors"

    # Describes Kubernetes source configuration.
    kubernetes:
      # Describes namespaces for every Kubernetes resources you want to watch or exclude.
      # These namespaces are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own namespaces object.
      namespaces: *k8s-events-namespaces

      # Describes events for every Kubernetes resources you want to watch or exclude.
      # These events are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own events object.
      events:
        - error

      # Describes the Kubernetes resources you want to watch.
      resources:
        - name: v1/pods
        - name: v1/services
        - name: networking.k8s.io/v1/ingresses
        - name: v1/nodes
        - name: v1/namespaces
        - name: v1/persistentvolumes
        - name: v1/persistentvolumeclaims
        - name: v1/configmaps
        - name: rbac.authorization.k8s.io/v1/roles
        - name: rbac.authorization.k8s.io/v1/rolebindings
        - name: rbac.authorization.k8s.io/v1/clusterrolebindings
        - name: rbac.authorization.k8s.io/v1/clusterroles
        - name: apps/v1/deployments
        - name: apps/v1/statefulsets
        - name: apps/v1/daemonsets
        - name: batch/v1/jobs
  'k8s-err-with-logs-events':
    displayName: "Kubernetes Errors for resources with logs"

    # Describes Kubernetes source configuration.
    kubernetes:
      # Describes namespaces for every Kubernetes resources you want to watch or exclude.
      # These namespaces are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own namespaces object.
      namespaces: *k8s-events-namespaces

      # Describes events for every Kubernetes resources you want to watch or exclude.
      # These events are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own events object.
      events:
        - error

      # Describes the Kubernetes resources you want to watch.
      resources:
        - name: v1/pods
        - name: apps/v1/deployments
        - name: apps/v1/statefulsets
        - name: apps/v1/daemonsets
        - name: batch/v1/jobs
        # replicasets excluded on purpose - to not show logs twice for a given e.g. deployment + replicaset

  'k8s-create-events':
    displayName: "Kubernetes Resource Created Events"

    # Describes Kubernetes source configuration.
    kubernetes:
      # Describes namespaces for every Kubernetes resources you want to watch or exclude.
      # These namespaces are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own namespaces object.
      namespaces: *k8s-events-namespaces

      # Describes events for every Kubernetes resources you want to watch or exclude.
      # These events are applied to every resource specified in the resources list.
      # However, every specified resource can override this by using its own events object.
      events:
        - create

      # Describes the Kubernetes resources you want to watch.
      resources:
        - name: v1/pods
        - name: v1/services
        - name: networking.k8s.io/v1/ingresses
        - name: v1/nodes
        - name: v1/namespaces
        - name: v1/configmaps
        - name: apps/v1/deployments
        - name: apps/v1/statefulsets
        - name: apps/v1/daemonsets
        - name: batch/v1/jobs

Run Botkube:

export BOTKUBE_SETTINGS_LOG_LEVEL=info
export BOTKUBE_SETTINGS_KUBECONFIG=$KUBECONFIG
export BOTKUBE_CONFIG_PATHS="$(pwd)/resource_config.yaml,$(pwd)/comm_config.yaml"
go run ./cmd/botkube/main.go

Trigger any event from command line, such as:

kubectl delete po -n kube-system -l k8s-app=metrics-server

You can also apply the following resources:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  minReadySeconds: 10 # by default is 0
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: registry.k8s.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "my-storage-class"
      resources:
        requests:
          storage: 1Gi

Or check global ones:

kubectl create ns foo

See the output:

Related issue(s)

Resolves #831

[pkosiec]

Now our integration tests have complexity of 37 🙈

[huseyinbabal]

- is this somehow not needed here? In your PR description, there is un expected dashes in Describe resource action which causes a wrong behavior. I removed dashes from curl brackets and it worked

[pkosiec]

It renders correctly in this place:

  global_config.yaml: |
    executors:
      kubectl-allow-all:
        # (...)

    actions:
      get-created-resource:
        # (...)

You're right though - in PR desc I had unnecessary dashes. I've actually fixed them before in the Helm chart values.yaml, but forgot to update (synchronize) them in PR desc. Sorry for that! Fixed now

[huseyinbabal]

Works as expected, I added only one minor comment.