Passive checks performance counters not showed in grafana

[ghost]

Hi!

Im using the plugin succesfully, even with multimetrics performance data.
But it only works with active checks.

When using passive checks, graphana does not show any data. The performance data is processed by Icinga2 because it is showed in the performance data table section in icingaweb.

What fix or extra configuration would be needed to generate graphs in graphana and make the grafana plugin show them?

Thanks!

[Mikesch-mp]

Hi,

can you enable debug output and configure proxy mode or direct mode please.
Then please post screenshot of debug output.

Regards,
Carsten

[ghost]

Hi Carsten

Not sure if this is what you need:

After sending this passive check result

echo "['date +%Y%m%d'] PROCESS_SERVICE_CHECK_RESULT;TEST-SERVER;TEST-PASSIVE-PERFORMANCE;0;Ok.... blabla|num=$(( ( RANDOM % 10 ) + 1 ));;;"

This logs are generated:

[2018-04-17 08:08:19 +0200] information/ExternalCommandListener: Executing external command: [20180417] PROCESS_SERVICE_CHECK_RESULT;TEST-SERVER;TEST-PASSIVE-PERFORMANCE;0;Ok.... blabla|num=6;;;
[2018-04-17 08:08:19 +0200] debug/DbEvents: add external command history
[2018-04-17 08:08:19 +0200] notice/ExternalCommandProcessor: Processing passive check result for service 'TEST-PASSIVE-PERFORMANCE'
[2018-04-17 08:08:19 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2018-04-17 08:08:19 +0200] notice/ApiListener: Relaying 'event::CheckResult' message
[2018-04-17 08:08:19 +0200] debug/DbEvents: add checkable check history for 'TEST-SERVER!TEST-PASSIVE-PERFORMANCE'
[2018-04-17 08:08:19 +0200] debug/InfluxdbWriter: Add to metric list: 'passive,hostname=TEST-SERVER,service=TEST-PASSIVE-PERFORMANCE,metric=num value=6 20180417'.
[2018-04-17 08:08:19 +0200] debug/InfluxdbWriter: Add to metric list: 'passive,hostname=TEST-SERVER,service=TEST-PASSIVE-PERFORMANCE acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max_check_attempts=1i,reachable=true,state=0i,state_type=1i 20180417'.

[Mikesch-mp]

Please enable the s"show debug" in the module configuration, see https://github.com/Mikesch-mp/icingaweb2-module-grafana/blob/master/doc/03-module-configuration.md#configuration-with-web-interface

Then take a screenshot of the passive service not showing any graph or empty graph, below the graph is all the information i want. Also please post your graphs.ini

[ghost]

My /etc/icingaweb2/modules/grafana/graphs.ini is empty.

[Mikesch-mp]

I will do some tests this evening.

[ghost]

Thank you very much

[Mikesch-mp]

I did some tests and it works flawless for me. How do you submit your data and do you use the provided default dashboards? Can you alo post the output of this influx query please

select * from passive  where hostname =~ /^TEST-SERVER$/  and service =~ /^TEST-PASSIVE-PERFORMANCE$/ limit 5

[ghost]

Im using a dashboard you provided me to allow multimetrics. See:
https://monitoring-portal.org/woltlab/index.php?thread/42595-grafana-not-showing-all-performance-counters-in-icinga2/

Anyway, I have just configured the "icinga2-default.json" and same problem.

Maybe the database is not being populated correctly? This is the output:

> select * from passive  where hostname =~ /^TEST-SERVER$/  and service =~ /^TEST-PASSIVE-PERFORMANCE$/
name: passive
-------------
time                    acknowledgement current_attempt downtime_depth  execution_time  hostname        latency max_check_attempts      metric  reachableservice                          state   state_type      value
20180222000000000                                                                       TEST-SERVER                                     num              TEST-PASSIVE-PERFORMANCE                         3
20180222000000000                                                                       TEST-SERVER                                     num              TEST-PASSIVE-PERFORMANCE                         72
20180222000000000       0               1               0               0               TEST-SERVER     0       1                               true     TEST-PASSIVE-PERFORMANCE 0       1
20180222000000000       0               1               0               0               TEST-SERVER     0       1                               true     TEST-PASSIVE-PERFORMANCE 0       1
20180417000000000                                                                       TEST-SERVER                                     num              TEST-PASSIVE-PERFORMANCE                         6
20180417000000000       0               1               0               0               TEST-SERVER     0       1                               true     TEST-PASSIVE-PERFORMANCE 0       1
20180423000000000                                                                       TEST-SERVER                                     num              TEST-PASSIVE-PERFORMANCE                         10
20180423000000000       0               1               0               0               TEST-SERVER     0       1                               true     TEST-PASSIVE-PERFORMANCE 0       1

This is the Icinga log when processing the passive command:

[2018-04-23 08:13:20 +0200] information/ExternalCommandListener: Executing external command: [20180423] PROCESS_SERVICE_CHECK_RESULT;TEST-SERVER;TEST-PASSIVE-PERFORMANCE;0;Ok.... blabla|num=9;;;
[2018-04-23 08:13:49 +0200] information/ExternalCommandListener: Executing external command: [20180423] PROCESS_SERVICE_CHECK_RESULT;TEST-SERVER;TEST-PASSIVE-PERFORMANCE;0;Ok.... blabla|num=9;;;
[2018-04-23 08:13:56 +0200] information/ExternalCommandListener: Executing external command: [20180423] PROCESS_SERVICE_CHECK_RESULT;TEST-SERVER;TEST-PASSIVE-PERFORMANCE;0;Ok.... blabla|num=1;;;

[ghost]

Hi again
Instead of using a test passive service, im using a real one that receives data every 2 minutes (instead of manually send data).
Now the graphs is being showed, but the value showed in the graph is not correct. For example, now its showing 70-80K instead of 370K.

Tried cleanup the series in influx but no success...

[Mikesch-mp]

If you want we can do remote session, so i can take a closer look at your problem. Just send me a email with your contact details and i will get back to you.

[ghost]

Thanks Mike.
But I think the problem is that influx is populating the database with the passive performance data with values that were reported 2 hours ago. I have checked the "debug/InfluxdbWriter" in /var/log/icinga2/debug.log, the values that were supposedly added to the metric list.

But the query directly to the metric is not logging the last cheked value but the value that was reported 2 hours ago.

Let me check more deeply. Ill keep you updated.

[ghost]

Sorry Mike. My fault when sending the timestamp.
Im not usually so inept jaja.

[ghost]

Thanks for your help!