initial prometheus2

[auhlig]

Similar to prometheus collector: 1h retention, no persistence. Also limited to discover endpoint, pods in namespaces ["k-staging","k-eu-nl"]. Reminder to self : Do not hardcode ns.
Should I add some metrics to the k*s controllers?
Also do we want node-, openstack exporter or kube-state-metrics? @BugRoger @databus23

If you want to take a look: Prometheus2 ( collector + frontend) is running in admin cluster.

[BugRoger]

Let's skip the collector. The main reason for the double-Prometheus setup was that we had to aggregate the CPU metrics with the 80 CPUs blades. Should not be a problem in this case anymore.

Similar to prometheus collector: 1h retention, no persistence

For the frontend we need persistence. Similar retention should be fine.

Also limited to discover endpoint, pods in namespaces ["k-staging","k-eu-nl"]

Just collect everything.

Should I add some metrics to the k*s controllers?

yes!

Also do we want node-, openstack exporter or kube-state-metrics?

Node and Kube-State-Metrics certainly won't hurt.

[BugRoger]

Would it be possible to ingress the Prometheus UIs to kubernikus.<region>.cloud.sap/prometheus?

[auhlig]

All done. Not sure if 9b81c6c works, but I'll give it a try.
I'll create a separate PR for the metrics and the exporter.

[BugRoger]

The config is pretty specific for the bare-metal control planes and most of the jobs don't apply here. But np, we can iterate and use it as a start.

I'm unsure about putting the chart into vendor. Wouldn't it be cleaner to just have it as a sub-chart? Having it in vendor suggests it's a generic chart that is reused multiple times.

[auhlig]

I just copied the config. Will remove and cleanup afterwards. Thought that might be easier.
Since it will eventually differ from the generic one it makes sense to mv it out of the vendor folder.

[BugRoger]

👍

[auhlig]

rebased and added kube-state-metrics (vendor) after all

[auhlig]

This is good to go from my side. Please merge @BugRoger. See: https://kubernikus.staging.cloud.sap/prometheus

[auhlig]

Do we want dedicated channels in slack, like #kubernikus-{critical|warning|info}? Or everything to the #kubernikus ch?

[BugRoger]

Very cool. Eventually we need to wall it off with authentication, but cool for now.

I'd prefer to have a similar channel setup. Makes it easier to tune the notification settings.

[auhlig]

am routing and channels setup done. invited you all. if no one objects, I'm going to merge this tomorrow.
regarding authentication: simple nginx reverse proxy or maia style with keystone integration?