Bump to kube stack 58.4.0 & prepare for deprecation of cattle.monitoring.io

[puffitos]

Motivation

To update the prometheus stack components, which were severely outdated and prepare the monitoring chart up to speed with the deprecation of the monitoring.cattle.io API group, which won't be used in the future (see caas-team/prometheus-auth#19)

Changes

• Changed RBAC to not use the deprecated apiGroup
• Radically minimized the values.yaml file, which had hundreds of default values from the kube-prometheus helm chart
• Possibility to make chart (a la caas-cluster-monitoring)
• Template README for the values
• Split the README files up, so they make for a little bit of easier reading

TODO

• Post a diff of the generated manifests with the old yaml and the current one (should be similar, according to the step by step tests and commits I've done)

Features

• Update to kube-prometheus-stack 58.4.0
• Update to grafana-7.3.9 chart
• New Prometheus Rules:
  • container-cpu-usage-seconds-total
  • container-memory-cache
  • container-memory-rss
  • container-memory-swap
  • container-memory-working-set-bytes
  • container-resource
  • pod-owner

A new, final target_matchers in the inhibit_rules for AlertmanagerConfig, to inhibit more Alerts:

- target_matchers:
  - alertname = InfoInhibitor

Deprecation Warning

** This project monitoring version won't work with older versions of caas-cluster-monitoring (< 0.0.13).**

The ServiceAccount isn't allowed to view prometheus.monitoring.cattle.io, but only prometheus.monitoring.coreos.com. This is part of a bigger deprecation we're doing to remove this non-existent API Group from our monitoring stack.

Smaller changes

• kubeprometheusstack/k8s-sidecar updated to 1.26.1
• probes are now selected as well based on the project name label
• field.cattle.io/projectId added as label to many resources

Other changes, not important for release notes

• #magic___^_^___line added to alertmanager config (autoformatting, nothing to worry about, was the same in caas-cluster-monitoring) -> we should move the template to a file.
• CPUThrottlingHigh, KubePersistentVolumeFillingUp, KubePersistentVolumeInodesFillingUp, rules aggregate on cluster as well (this may break the rule, nothing we can do here)

The following prom rules were also changed:

• PrometheusNotIngestingSamples
• AlertmanagerFailedToSendAlerts
• AlertmanagerClusterFailedToSendAlerts

Tests done

A previous version (which was exactly similar to this one) is already running in our dev clusters. Still open:

• E2E deployment of this version, so assert compatibility with new prometheus-auth

[y-eight]

LGTM