Introducing a new component monitoring-integration(1st milestone)

specs/monitoring_integration.adoc

@@ -0,0 +1,263 @@
+= Spec - Introducing a new component monitoring-integration(1st milestone)
+
+A new component called monitoring-Integration is introduced which integrates Grafana
+with tendrl core.
+
+
+== Problem description
+
+The new monitoring-integration component would primarily focus on dynamic
+configuration of grafana, based on the topology changes in etcd. Some of these
+responsibilities would be:
+
+* Monitoring-integration should create default dashboards in grafana using predefined json.
+  The dashboard json contains details regarding panels, metrics and alerts.
+* Configuring grafana authentication and authorization (https://github.com/Tendrl/specifications/pull/205),
+  based on changes in the primary tendrl AA scheme.
+* Keeping track of the topology changes in etcd to configure the grafana dashboards
+  to ensure that the correct nodes are part of the respective cluster dashboards.
+  (Not planned yet, for 1st milestone grafana is configured using some default
+  configuration from monitoring-integration)
+
+
+== Use Cases
+
+For 1st milestone:
+
+* Monitoring-Integration will create default dashboards in grafana.
+
+For 2nd milestone:
+
+* Grafana alerting events should be notified via tendrl.
+* Monitoring-integration should be able to configure grafana based on topology
+  changes in etcd.
+
+For 3rd milestone:
+
+* User should be able access grafana dashboard via tendrl user interface using
+  tendrl authentication.
+
+== Proposed change
+
+=== Why monitoring-integration?
+
+Grafana is used for visualizing time series data in graphs. Based on the metrics
+from datasource it will create graphs and display. It provides a rich set of tools
+to display monitoring and analytics using time series data. monitoring-integration
+component would primarily focus on dynamic configuration of grafana, based on
+the topology changes in etcd.
+
+Monitoring-integration and grafana server are installed via tendrl-ansible in server machine
+(for 1st milestone it is installed manually). There is just one instance of monitoring-integration
+and grafana installed across all clusters. For now when monitoring-integration starts
+then it will create gafana dashboards using some predefined json files.
+(Complelete flow structure for monitoring-integration is out of scope of this specification).
+
+For each grafana dashboard different predefined json is maintained in monitoring-integration.
+The json is sent to grafana server as http api request.
+
+JSON Body schema:
+
+* dashboard – The complete dashboard model, id = null to create a new dashboard
+* overwrite – Set to true if you want to overwrite existing dashboard with newer
+  version or with same dashboard title.
+* message - Set a commit message for the version history.
+
+```
+Example dashboard json for creating a new dashboard or updating an existing dashboard.
+
+{
+  "dashboard": {
+    "id": null,
+    "title": "Production Overview",
+    "tags": [ "templated" ],
+    "timezone": "browser",
+    "rows": [
+      {
+      }
+    ],
+    "schemaVersion": 6,
+    "version": 0
+  },
+  "overwrite": false
+}
+```
+
+Monitoring-integration is enabling anonymous access to create dashboards and
+datasource by changing grafana configuration file. It Setting the organization name
+that should be used for anonymous users.
+
+
+== Alternatives
+
+None
+
+
+== Data model impact:
+
+None
+
+
+== Impacted Modules:
+
+=== Tendrl API impact:
+
+None
+
+=== Notifications/Monitoring impact:
+
+None
+
+=== Tendrl/common impact:
+
+None
+
+=== Tendrl/node_agent impact:
+
+None
+
+=== Sds integration impact:
+
+None
+
+=== Tendrl/monitoring-integration impact:
+
+Introduce a new component monitoring-integration in tendrl.
+
+Monitoring integration should have all predefined json files and configuration files
+for grafana. When monitoring-integration starts it will create the grafana dashboard
+and datasource using predefined jsons via http api request.
+
+```
+Grafana APIs used in monitoring-integration:
+
+1) Dashboard API:
+    To creates a new dashboard or update an existing dashboard.
+       POST -  /api/dashboards/db
+        {
+          "dashboard": {
+          "id": null,
+          "title": "Brick utilization",
+          "tags": [ "" ],
+          "timezone": "",
+          "rows": [
+          {
+          }
+          ],
+          "schemaVersion": 6,
+          "version": 0
+          },
+          "overwrite": false
+        }
+2) Data source API
+    To create data source
+      POST /api/datasources
+      {
+        "name":"datasource_name",
+        "type":"graphite",
+        "url":"",
+        "access":"proxy",
+        "basicAuth":false
+      }
+3) Create Organisation
+    To create organisation
+      POST /api/orgs
+      {
+        "name":"New Org."
+      }
+```
+
+
+=== Security impact:
+
+None
+
+=== Other end user impact:
+
+User can access the grafana dashboards using tendrl authorization only.
+
+=== Performance impact:
+
+None
+
+=== Other deployer impact:
+
+Monitoring-integration needs grafana packages as dependency.
+
+=== Developer impact:
+
+Create a new component called monitoring-integration and implement its functionalities.
+
+== Implementation:
+
+Implementation details for milestone 1:
+
+* Create a new tendrl component called "monitoring-integration".
+* Create a directory called "etc" to store configuration files:
+    1. grafana/grafana.ini
+    2. tendrl/monitoring-integration/monitoring-integration.conf.sample
+    3. tendrl/monitoring-integration/grafana/dashboards/tendrl-gluster-at-a-glance.json
+    4. tendrl/monitoring-integration/grafana/dashboards/tendrl-gluster-bricks.json
+    5. tendrl/monitoring-integration/grafana/dashboards/tendrl-gluster-hosts.json
+    6. tendrl/monitoring-integration/grafana/dashboards/tendrl-gluster-volumes.json
+    7. tendrl/monitoring-integration/graphite/carbon.conf.sample
+    8. tendrl/monitoring-integration/graphite/graphite-web.conf.sample
+* Create tendrl-monitoring-integration.spec file for packaging.
+* Create a tox.ini file to run unit test cases.
+* Create a setup.py file for package installation.
+* Create a version.py file to specify monitoring-integration component version.
+* Create a check_commit_msg.py file to check the commit messages by travis.
+* Create .travis.yml file for Travis CI service.
+* Create a new directory called tendrl.
+* Create a new directory called monitoring_integration inside tendrl directory.
+* Create new file "__init__.py" in tendrl/monitoring_integration and wite a loggic
+  to initiate the grafana dashboard and datasource creation process.
+* Create a new directory called grafana in tendrl/monitoring_integration.
+* Create a file called dashboard.py in tendrl/monitoring_integration to create grafana
+  dashboard.
+* Create a file datasource.py in tendrl/monitoring_integration to create grafana
+  datasource.
+* Create a file called utils.py in tendrl/monitoring_integration/grafana.
+* utils functions are used to make different api calls for create dashboard
+  and datasource in grafana.
+
+Workflow planned for first milestone is:
+
+When monitoring-integration starts then it is configured from a configuration file
+"/etc/tendr/monitoring_integration/monitoring-integration.conf".
+Then it will create dashboards and datasources in grafana using predefined
+json files from tendrl/monitoring_integration/grafana directory with the help of util functions. When
+monitoring_integration restart it will compare existing dashboards in grafana with predefined
+dashboards in monitoring integration, if any dashboard is not present in grafana then it will create a
+new dashboard otherwise it will skip.
+
+
+== Assignee(s):
+
+@GowthamShanmugam
+
+@rishubhjain
+
+== Work Items:
+
+https://github.com/Tendrl/specifications/issues/179
+
+
+== Testing:
+
+Check grafana dashboards are configured and displayed correctly based on topologies
+from etcd.
+
+
+== Documentation impact:
+
+None
+
+
+== References:
+
+https://github.com/Tendrl/specifications/pull/198
+
+https://github.com/Tendrl/specifications/pull/205
+
+https://github.com/Tendrl/specifications/pull/171