apps sc: made alert runbooks configurable

[davidumea]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

Makes it possible to configure runbooks, can be configured on an alert group level or per individual alert.

Current override priority:

1. Use individual alert runbook override if set
2. Use group alert runbook override if set
3. Use upstream alert runbook by default if exists
4. Runbook is set to "Missing runbook" if there's no upstream runbook and nothing is configured.

• Fixes https://github.com/elastisys/ck8s-issue-tracker/issues/423

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[anders-elastisys]

Nice work, I really like this change. Added some comments, otherwise it looks good

[anders-elastisys]

Nice 👍

[viktor-f]

Nice work! I think that this solution ended up looking very clean.

[viktor-f]

I think that this description is nice, but I see that this one is the only one with the examples. It would be nice if we had the examples for all groups, not just alertmanager, but I also don't want us to duplicate this text for each.

Suggestion: Could you add this to a template instead and add a reference here? Then we can get it to all without having to duplicate the text. You could have two templates, one with the text about using upstream runbook by default and one without. What do you think about this? Did you already have some similar plan?

[Zash]

Are all these objects different? I wonder if there's any way to have reusable common bits?

[viktor-f]

The names are a bit different, but I think you could just create an example that is generic and use that for all alerts.

[Zash]

Hm, but for validation? I guess "additionalProperties":{"type":"string"} might be good enough

[aarnq]

Please add a $def and reuse it for all, and give group a specific one as it has special handling, else @Zash suggestion is the way to go for map[string]string for the alerts themselves.

[davidumea]

👍 I think this works, at least it validates

c835a3c

[viktor-f]

Is this still used?

compliantkubernetes-apps/helmfile.d/charts/prometheus-alerts/values.yaml

Line 78 in 348bc94

runbookUrl: "https://runbooks.prometheus-operator.dev/runbooks/"

If not, then I think it can be removed.

[davidumea]

I can remove it

[davidumea]

a9550e5

[aarnq]

Define this object, also you should move the $def under runbookUrls because it is not going to be used anywhere else.

[davidumea]

Define this object

I don't think we should define the object (each alert), to avoid default config clutter.

Or am I missing the point?

[aarnq]

You should define group as a string and define it, then define alerts under additionalProperties also as a string.

[davidumea]

Do you want me to define every single alert?

[davidumea]

Is this how you meant? 🙂 6eb927c