Consider Grafana or Netdata for system metrics

[RobHooper]

Grafana is a web interface for visualising data, perfect for our Prometheus system metrics.

Grafana will make it much easier to use and interpret server monitoring, including:

• A time picker updating all graphs at once
• A server picker to filter all graphs to one server at a time
• Simple URL, the queries are stored server-side so we will no longer need long complex custom links.
• A simpler interface will make it easier to use in an emergency.

To do:

• Deploy Grafana
• Configure user accounts
• Integrate with Prometheus data source
• Create custom dashboard

We already have custom queries written for the current monitor setup, we can re-use these in the new dashboard.

The Prometheus server will be best to host this assuming it has spare system resources.

[RobHooper]

We now have Netdata installed on each server as part of Dogsbody's monitoring stack.
Netdata could replace Prometheus and the above proposed Grafana install - needs investigating.

[jpmckinney]

Good reminder. Yes, it sounds like it will be simpler to switch to netdata than to setup (and maintain) Grafana.

[jpmckinney]

Noting that if RabbitMQ's management interface metrics are removed, then we would need Prometheus/Grafana. https://www.rabbitmq.com/docs/prometheus

Edit: If we switch to Netdata, we should add our our Salt states to install it (Dogsbody installed and configured it manually).

[jpmckinney]

I notice that Netdata uses GBs of data on e.g. ocp18 at /var/cache/netdata. Not sure if this can be avoided?

[RobHooper]

I notice that Netdata uses GBs of data on e.g. ocp18 at /var/cache/netdata. Not sure if this can be avoided?

This should be configurable in Netdata setting a limit to the amount of local storage it can use.
I would have expected monitoring data to be sent and stored in Netdata Cloud so it seems strange it is using so much.