Merge pull request Tendrl#218 from GowthamShanmugam/monitoring_integr…

…ation

Introducing a new component monitoring-integration(1st milestone)

specs/monitoring_integration.adoc

@@ -0,0 +1,233 @@
+= Spec - Introducing a new component monitoring-integration(1st milestone)
+
+A new component called monitoring-Integration is introduced which integrates Grafana
+with the tendrl core.
+
+
+== Problem description
+
+The new monitoring-integration component would primarily focus on configuration of
+grafana. Monitoring-integration should create default dashboards in grafana using
+predefined JSON.
+
+
+== Use Cases
+
+* Monitoring-Integration will create default dashboards in grafana.
+
+
+== Proposed change
+
+=== Why monitoring-integration?
+
+Grafana is used for visualizing time series data in graphs. Based on the metrics
+from data-source it will create graphs and display. It provides a rich set of tools
+to display monitoring and analytics using time series data. monitoring-integration
+component would primarily focus on configuration of grafana.
+
+Monitoring-integration and grafana server are installed manually in the server machine.
+There is just one instance of monitoring-integration and grafana installed across
+all clusters. For now when monitoring-integration starts then it will create gafana
+dashboards using some predefined JSON files.
+(Complete flow structure for monitoring-integration is out of scope of this specification).
+
+For each grafana dashboard different predefined JSON is maintained in monitoring-integration.
+The JSON is sent to grafana server as HTTP API request.
+
+JSON Body schema:
+
+* dashboard – The complete dashboard model, id = null to create a new dashboard
+* overwrite – Set to true if you want to overwrite existing dashboard with newer
+  version or with same dashboard title.
+* message - Set a commit message for the version history.
+
+```
+Example dashboard JSON for creating a new dashboard or updating an existing dashboard.
+
+{
+  "dashboard": {
+    "id": null,
+    "title": "Production Overview",
+    "tags": [ "templated" ],
+    "timezone": "browser",
+    "rows": [
+      {
+      }
+    ],
+    "schemaVersion": 6,
+    "version": 0
+  },
+  "overwrite": false
+}
+```
+
+Monitoring-integration is enabling anonymous access to create dashboards and
+data-source by changing grafana configuration file. It Setting the organization name
+that should be used for anonymous users.
+
+
+== Alternatives
+
+None
+
+
+== Data model impact:
+
+None
+
+
+== Impacted Modules:
+
+=== Tendrl API impact:
+
+None
+
+=== Notifications/Monitoring impact:
+
+None
+
+=== Tendrl/common impact:
+
+None
+
+=== Tendrl/node_agent impact:
+
+None
+
+=== Sds integration impact:
+
+None
+
+=== Tendrl/monitoring-integration impact:
+
+Introduce a new component monitoring-integration in tendrl.
+
+Monitoring integration should have all predefined json files and configuration files
+for grafana. When monitoring-integration starts it will create the grafana dashboard
+and datasource using predefined JSON via HTTP API request.
+
+```
+Grafana APIs used in monitoring-integration:
+
+1) Dashboard API:
+    To create a new dashboard or update an existing dashboard.
+       POST -  /api/dashboards/db
+        {
+          "dashboard": {
+          "id": null,
+          "title": "Brick utilization",
+          "tags": [ "" ],
+          "timezone": "",
+          "rows": [
+          {
+          }
+          ],
+          "schemaVersion": 6,
+          "version": 0
+          },
+          "overwrite": false
+        }
+2) Data source API
+    To create data source
+      POST /api/datasources
+      {
+        "name":"datasource_name",
+        "type":"graphite",
+        "url":"",
+        "access":"proxy",
+        "basicAuth":false
+      }
+3) Create Organization
+    To create organisation
+      POST /api/orgs
+      {
+        "name":"New Org."
+      }
+```
+
+
+=== Security impact:
+
+None
+
+=== Other end user impact:
+
+Users can access the grafana dashboards using tendrl authorization only.
+
+=== Performance impact:
+
+None
+
+=== Other deployer impact:
+
+Monitoring-integration needs grafana packages as a dependency.
+
+=== Developer impact:
+
+Create a new component called monitoring-integration and implement its functionalities.
+
+== Implementation:
+
+Implementation details for milestone 1:
+
+* Create a new tendrl component called "monitoring-integration".
+* Create a directory called "etc" to store configuration files:
+    1. grafana/grafana.ini
+    2. tendrl/monitoring-integration/monitoring-integration.conf.sample
+* Copy the monitoring-integration.conf.sample file into
+    /etc/tendrl/monitoring-integration/monitoring-integration.conf".
+* Create a tox.ini file to run unit test cases.
+* Create a setup.py file for package installation.
+* Create a version.py file to specify monitoring-integration component version.
+* Create a check_commit_msg.py file to check the commit messages by travis.
+* Create .travis.yml file for Travis CI service.
+* Create a new directory called tendrl.
+* Create a new directory called monitoring_integration inside tendrl directory.
+* Create new file "__init__.py" in tendrl/monitoring_integration and write a loggic
+  to create a grafana dashboard and data-source.
+* Create a new directory called grafana in tendrl/monitoring_integration.
+* All predefined JSON files for creating a grafana dashboards are placed in
+  tendrl/monitoring_integration/grafana.
+* Create a file called utils.py in tendrl/monitoring_integration/grafana.
+* utils functions are used to make different api calls for create dashboard
+  and data-source in grafana.
+
+Workflow planned for The first milestone is:
+
+When monitoring-integration starts then it is configured from a configuration file
+"/etc/tendr/monitoring_integration/monitoring-integration.conf".
+Then it will create dashboards and data-sources in grafana using predefined
+JSON files from etc/tendrl/monitoring_integration/grafana directory with the help of util functions. When
+monitoring_integration restart it will compare existing dashboards in grafana with predefined
+dashboards in monitoring integration, if any dashboard is not present in grafana then it will create a
+new dashboard otherwise it will skip.
+
+
+== Assignee(s):
+
+@GowthamShanmugam
+
+@rishubhjain
+
+== Work Items:
+
+https://github.com/Tendrl/specifications/issues/179
+
+
+== Testing:
+
+Check dashboards and data-source are created successfully in grafna. And it is accessible
+via anonymous access.
+
+== Documentation impact:
+
+None
+
+
+== References:
+
+https://github.com/Tendrl/specifications/pull/198
+
+https://github.com/Tendrl/specifications/pull/205
+
+https://github.com/Tendrl/specifications/pull/171