Introducing a new component monitoring-integration(1st milestone) #218
Conversation
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
|
@shtripat @r0h4n @brainfunked @nnDarshan @rishubhjain @anivargi please review |
specs/monitoring_integration.adoc
Outdated
| configuration of grafana, based on the topology changes in etcd. Some of these | ||
| responsibilities would be: | ||
|
|
||
| * monitoring-integration should to store predefined json’s for dashboards, panels and alerts. |
There was a problem hiding this comment.
s/should to store/should store/
There was a problem hiding this comment.
i will change it
specs/monitoring_integration.adoc
Outdated
| * monitoring-integration should to store predefined json’s for dashboards, panels and alerts. | ||
| * Keeping track of the topology changes in etcd to configure the grafana dashboards | ||
| to ensure that the correct nodes are part of the respective cluster dashboards. | ||
| * Configuring grafana authentication and authorisation (https://github.com/Tendrl/specifications/pull/205), |
There was a problem hiding this comment.
s/authorisation/authorization/
specs/monitoring_integration.adoc
Outdated
|
|
||
| == Use Cases | ||
|
|
||
| * Monitoring-integration should able to configure grafana dashboard and panels |
specs/monitoring_integration.adoc
Outdated
|
|
||
| * Monitoring-integration should able to configure grafana dashboard and panels | ||
| and alerts. | ||
| * User should able access grafana dashboard via tendrl user interface using |
There was a problem hiding this comment.
s/should able/ should be able to/
specs/monitoring_integration.adoc
Outdated
| 4. Dashboard | ||
| 5. Dashboard version | ||
| 6. Data source | ||
| 7. Organisation |
specs/monitoring_integration.adoc
Outdated
| Introduced a new component monitoring integration in tendrl. | ||
|
|
||
| Monitoring integration should have all predefined json files. When monitoring | ||
| integration stats it should start the script to configure the grafana environment |
specs/monitoring_integration.adoc
Outdated
| * Create a new tendrl component called monitoring-integration | ||
| * Create a directory called grafana to place all predefined json files and configuration | ||
| files for grafana. | ||
| * Create a files in grafana directory to configure the grafana environment using |
There was a problem hiding this comment.
s/Create a files/ Create files/
specs/monitoring_integration.adoc
Outdated
| predefined jsons. | ||
| * Create a grafana configuration file called grafana.conf.yml | ||
| * create a graphite configuration file called graphite.conf.yml | ||
| * Create a directory called server to create an endpoint for receive grafana |
specs/monitoring_integration.adoc
Outdated
|
|
||
| == Assignee(s): | ||
|
|
||
| Gowtham S & Rishub jain |
specs/monitoring_integration.adoc
Outdated
|
|
||
| == Work Items: | ||
|
|
||
| None |
There was a problem hiding this comment.
Put the specific issues links here
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
GowthamShanmugam
force-pushed
the
monitoring_integration
branch
from
b582a02 to
0818cdf
Compare
|
@shtripat changes are done |
|
@brainfunked @r0h4n @nnDarshan @anivargi please review |
specs/monitoring_integration.adoc
Outdated
| predefined jsons. | ||
| * Create a grafana configuration file called grafana.conf.yml | ||
| * create a graphite configuration file called graphite.conf.yml | ||
| * Create a directory called server to create an endpoint for receive grafana | ||
| * Create a directory called server to create an endpoint for receiving grafana | ||
| alert events and convert that alert events into tendrl alert condition and pass the |
There was a problem hiding this comment.
Where are we going to store the default values like:
default panel, dashboard, database name, and other details.
There was a problem hiding this comment.
it is in predefined json only ...is hat correct @anivargi
specs/monitoring_integration.adoc
Outdated
| responsibilities would be: | ||
|
|
||
| * monitoring-integration should store predefined json’s for dashboards, panels and alerts. | ||
| * Keeping track of the topology changes in etcd to configure the grafana dashboards |
There was a problem hiding this comment.
How this is done? is that done as part of this spec?
There was a problem hiding this comment.
No, this is not for this milestone. we are going to use default values
There was a problem hiding this comment.
@anivargi need your comment
There was a problem hiding this comment.
Regardless of when it is done, please add the details in the spec
There was a problem hiding this comment.
i have mentioned that in under second milestone section
https://github.com/Tendrl/specifications/pull/218/files#diff-339c615a84970e1b496c5433f62b799f
Flow structure for that may be covered in different spec in second milestone
specs/monitoring_integration.adoc
Outdated
|
|
||
| == Use Cases | ||
|
|
||
| * Monitoring-integration should be able to configure grafana dashboard and panels |
There was a problem hiding this comment.
As per yesterday's discussion, I think this is the only use-case addressed in first milestone. State this explicitly on the spec issue
There was a problem hiding this comment.
ok i will add this point in spec
specs/monitoring_integration.adoc
Outdated
|
|
||
| Grafana is used for visualizing time series data in graphs. Based on the metrics | ||
| from datasource it will create graphs and display. Grafana supports different kinds | ||
| of http api's to configure its environment. |
There was a problem hiding this comment.
What are grafana APIs used as part of this spec? Can you list them and purpose of each?
There was a problem hiding this comment.
@rishubhjain can you list down the API's that you have used to configure grafana?
There was a problem hiding this comment.
i will add that
There was a problem hiding this comment.
I feel @cloudbehl needed actual end-points here
specs/monitoring_integration.adoc
Outdated
| keep track of the topology changes in etcd to configure the grafana dashboards. | ||
|
|
||
| Monitoring integration should have all grafana related packages as dependency. | ||
| When cluster is created or imported, the flow will install the monitoring integration |
There was a problem hiding this comment.
Monitoring integration is installed when the cluster imported/created? So you mean it is per cluster? I don't think this is correct. Per my understanding this service(grafana as well) is per tendrl instance(just like performance monitoring) and dash board poulation should be done as part of import
If that is the case, how monitoring integration is installed?(I assume via tendrl-ansible). Please provide the complete workflow here
@anivargi , @brainfunked , please comment
There was a problem hiding this comment.
i got your point, i need more info on this flow
There was a problem hiding this comment.
There is just 1 instance of monitoring-integration + grafana installed across all clusters.
specs/monitoring_integration.adoc
Outdated
| Introduce a new component monitoring integration in tendrl. | ||
|
|
||
| Monitoring integration should have all predefined json files. When monitoring | ||
| integration starts it should start the script to configure the grafana environment |
There was a problem hiding this comment.
What happens when someone restarts monitoring-integration?
There was a problem hiding this comment.
I would suggest to maintain a state within monitoring-integration which mentions if the grafana configuration is already done or not. If its already done, dont do it again. May be an entry in etcd would work.
I remember this was taken in performance-monitoring, where it used to execute configure-collectd on nodes only once. Check the logic from there as well.
There was a problem hiding this comment.
@nthomas-redhat for first milestone we are creating monitoring-integration like this, if monitoring integration start then it will start configure grafana. we are not decided complete flow of monitoring integration yet
There was a problem hiding this comment.
@shtripat @nthomas-redhat For the 4th drop the role of monitoring integration is just to create default dashboards.A check already exists which does not create/update the dashboards again if the dashboards already exists.
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
specs/monitoring_integration.adoc
Outdated
|
|
||
| * Monitoring-integration should be able to configure grafana dashboard and panels | ||
| and alerts. | ||
| * User should be able access grafana dashboard via tendrl user interface using |
There was a problem hiding this comment.
I think this is not a part of 1rst Milestone
There was a problem hiding this comment.
Is gafana access is always anonymous?
specs/monitoring_integration.adoc
Outdated
| } | ||
| ``` | ||
|
|
||
| For each grafana configuration different predefined json is maintained in monitoring |
There was a problem hiding this comment.
In my opinion instead of grafana configuration it should be grafana dashboard
specs/monitoring_integration.adoc
Outdated
| Monitoring integration also monitors the alerting event from grafana and notify it | ||
| to user via tendrl (https://github.com/Tendrl/specifications/pull/198). | ||
|
|
||
| monitoring integration passes dashboard related metrics in dashboard json. Alert |
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
|
@r0h4n @shtripat @nthomas-redhat @rishubhjain @brainfunked @anivargi please review |
specs/monitoring_integration.adoc
Outdated
| configuration of grafana, based on the topology changes in etcd. Some of these | ||
| responsibilities would be: | ||
|
|
||
| * monitoring-integration should store predefined json’s for dashboards, panels and alerts. |
There was a problem hiding this comment.
monitoring-integration - > Monitoring-Integration
There was a problem hiding this comment.
ah i will change this
There was a problem hiding this comment.
Monitoring Integration stores json for default dashboard which contains details about panels, alerts and metrics
Signed-off-by: GowthamShanmugam <[email protected]>
specs/monitoring_integration.adoc
Outdated
| configuration of grafana, based on the topology changes in etcd. Some of these | ||
| responsibilities would be: | ||
|
|
||
| * Monitoring-integration should store predefined json’s for dashboards, panels and alerts. |
There was a problem hiding this comment.
Monitoring-integration should create default dashboards in grafana using predefined json.
The dashboard json contains details regarding panels.,metrics and alerts.
specs/monitoring_integration.adoc
Outdated
|
|
||
| For 1st milestone: | ||
|
|
||
| * Monitoring-integration should be able to configure grafana dashboard and panels. |
There was a problem hiding this comment.
Monitoring-Integration will create default dashboards in grafana
specs/monitoring_integration.adoc
Outdated
| For 1st milestone: | ||
|
|
||
| * Monitoring-integration should be able to configure grafana dashboard and panels. | ||
| * Allow tendrl users to access grafana using anonymous authorization. |
There was a problem hiding this comment.
I think this is not included in the 1st milestone
specs/monitoring_integration.adoc
Outdated
| * Grafana alerting events should be notified via tendrl. | ||
| * Monitoring-integration should be able to configure grafana based on topology | ||
| changes in etcd. | ||
| * User should be able access grafana dashboard via tendrl user interface using |
There was a problem hiding this comment.
based on that only authentication logic will change here
There was a problem hiding this comment.
This is a part of 3rd Milestone https://github.com/Tendrl/specifications/milestone/1
specs/monitoring_integration.adoc
Outdated
| 11. Others | ||
|
|
||
| Monitoring integration is the component which will be used to configure the grafana | ||
| environment when it is started. Monitoring integration is going to have all json |
There was a problem hiding this comment.
Monitoring Integration will contain json related to default dashboards
There was a problem hiding this comment.
no this statement looks like static for 1st milestone
Signed-off-by: GowthamShanmugam <[email protected]>
Signed-off-by: GowthamShanmugam <[email protected]>
Signed-off-by: GowthamShanmugam <[email protected]>
Signed-off-by: GowthamShanmugam <[email protected]>
GowthamShanmugam
force-pushed
the
monitoring_integration
branch
from
0d36a76 to
d68ca11
Compare
|
@r0h4n @anivargi @rishubhjain @nthomas-redhat updated all changes |
specs/monitoring_integration.adoc
Outdated
| @@ -0,0 +1,396 @@ | |||
| = Spec - Introducing a new component monitoring-integration | |||
|
|
|||
| A new component called Monitoring-Integration is introduced which integrates Grafana | |||
There was a problem hiding this comment.
s/Monitoring-Integrtaion/monitoring-integration/
specs/monitoring_integration.adoc
Outdated
|
|
||
| Grafana is used for visualizing time series data in graphs. Based on the metrics | ||
| from datasource it will create graphs and display. Grafana supports different kinds | ||
| of http api's to configure its environment. |
There was a problem hiding this comment.
I feel @cloudbehl needed actual end-points here
specs/monitoring_integration.adoc
Outdated
| 10. Snapshot | ||
| 11. Others | ||
|
|
||
| Monitoring Integration will contain json related to dashboards. Monitoring integration should |
There was a problem hiding this comment.
Everywhere use the name as monitoring-integration
specs/monitoring_integration.adoc
Outdated
| Monitoring integration and grafana server are installed via tendrl-ansible in server machine. | ||
| There is just one instance of monitoring-integration and grafana installed across all clusters. | ||
| For now when monitoring integration starts then it will create gafana dashboards | ||
| using some predefined json files (Entire flow is not planned yet). |
There was a problem hiding this comment.
Instead of saying this may be you can say the flow structure is out of scope of this specification.
specs/monitoring_integration.adoc
Outdated
|
|
||
| The json is sent to grafana server as http api request. | ||
|
|
||
| JSON Body schema: |
There was a problem hiding this comment.
If its a JSON payload, can you add a sample JSON payload data itself?
specs/monitoring_integration.adoc
Outdated
| Grafana APIs used in monitoring integration: | ||
|
|
||
| 1) Dashboard API: | ||
| To creates a new dashboard or updates an existing dashboard. |
specs/monitoring_integration.adoc
Outdated
|
|
||
| === Tendrl/node_agent impact: | ||
|
|
||
| Cluster creation or import cluster flow should install the monitoring-integration |
There was a problem hiding this comment.
Shouldn't monitoring-integration be installed by tendrl-ansible in the server node already and during create/import only the grafana configurations should be deployed and enabled?
I understand this might not be planned for milestone-1 though :)
There was a problem hiding this comment.
ya i will change it
There was a problem hiding this comment.
It is not for miles stone one
GowthamShanmugam
force-pushed
the
monitoring_integration
branch
from
24a3d7f to
c46604c
Compare
GowthamShanmugam
force-pushed
the
monitoring_integration
branch
from
c46604c to
5f4e940
Compare
Signed-off-by: GowthamShanmugam <[email protected]>
GowthamShanmugam
force-pushed
the
monitoring_integration
branch
from
5f4e940 to
6e9fcf1
Compare
specs/monitoring_integration.adoc
Outdated
|
|
||
| * User should be able access grafana dashboard via tendrl user interface using | ||
| tendrl authentication. | ||
|
|
There was a problem hiding this comment.
When you planing to handle the updates?for ex updates to json files during yum update?
Also monitoring-integration as a service?
There was a problem hiding this comment.
These things are planned for 2nd milestone, For now monitoring-integration is just a one time manual run.
It is now it work just like a script.
So @TimothyAsir is working on .spec after that only we will change monitoring-integration as service
Signed-off-by: GowthamShanmugam <[email protected]>
GowthamShanmugam
changed the title
|
@r0h4n please review |
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
specs/monitoring_integration.adoc
Outdated
| == Problem description | ||
|
|
||
| The new monitoring-integration component would primarily focus on dynamic | ||
| configuration of grafana, based on the topology changes in etcd. Some of these |
There was a problem hiding this comment.
Are we doing any dynamic configuration based on the topology changes in milestone 1?
There was a problem hiding this comment.
no i will remove it
specs/monitoring_integration.adoc
Outdated
|
|
||
| * Monitoring-Integration will create default dashboards in grafana. | ||
|
|
||
| For 2nd milestone: |
There was a problem hiding this comment.
Heading talks about milestone-1 and you have details about milestone-2 & 3 here. As we talked about in various discussions, take this out from this spec, file specific issues for milestone-2 & 3 items. We need milestone specific specs.
There was a problem hiding this comment.
I will remove it
specs/monitoring_integration.adoc
Outdated
| The dashboard json contains details regarding panels, metrics and alerts. | ||
| * Configuring grafana authentication and authorization (https://github.com/Tendrl/specifications/pull/205), | ||
| based on changes in the primary tendrl AA scheme. | ||
| * Keeping track of the topology changes in etcd to configure the grafana dashboards |
There was a problem hiding this comment.
Keeping track of the topology changes is something which is done in milestone-1?
There was a problem hiding this comment.
i will remove it
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
tendrl-bug-id: Tendrl#179 Signed-off-by: GowthamShanmugam <[email protected]>
|
@nthomas-redhat spec is modified to 1sr milestone please review |
tendrl-bug-id: #179
Signed-off-by: GowthamShanmugam [email protected]