Support Sensu

[sstarcher]

We currently use sensu for all of our alerting needs and routing.

[JulienChampseix]

+1

[nathanielc]

Could either of you provide and example TICKscript of what sending alert data to Sensu would look like? Don't worry about implementation just how you would expect it to look in the script. Thanks

[sstarcher]

I'm not 100% sure what you are asking for so if I don't answer it correctly let me know.

Minimal data would look like the following.

{
   "name":"{{ .Env.NAME }}",
   "source":"kapacitor",
   "output":"{{ .Env.OUTPUT }}",
   "status": {{ .Env.STATUS }},
}

• Output can be any string
• Name would be a unique name for the alert - /^[\w\.-]+$/
• Status integer - 0=OK, 1=warn, 2=critical, 3+ unknown

https://sensuapp.org/docs/0.21/clients

Other configuration information required

• HOST
• PORT

TICKscript?

    stream...
         .alert()
             .sensu()

[nathanielc]

@sstarcher That is what I was looking for thanks

Do you expect Kapacitor to act as a sensu client (heartbeats etc) or just send events over to sensu via the API and let it do the rest?

[sstarcher]

I would say the easiest implementation would be to not send heartbeats. My main goal would be to have all of my alerting outbound in one place. Kapacitor would detect the problem and instead of configuring Kapacitor to send to slack/pagerduty etc. We would instead using our existing sensu setup that all of our teams have configured routing and alerting rules already for.

[jcmartins]

But with telegraf in all clients sending data to Influxdb and make alerts/checks in Kapacitor, you don't need use sensu anymore. ;-)

[sstarcher]

@jcmartins I'll assume sarcasm ;)

As sensu has the capabilities to alert on much more than metrics.

[jcmartins]

@sstarcher In kapacitor you can have alert like sensu(warning, critical, ok) but I agree with you kapacitor and telegraf don't have mature and same features yet.
I'm using sensu+influxdb+grafana a lot with victorops on 400 ec2 aws instances and 50 RDS, but I'm doing some tests here and I'm surprised at kapacitor + Telegraf can do.
I'll need to create/customize some plugins for Telegraf as I have done for Sensu.
Maybe I'll use it on my dev/uat environments in the version 0.4 February 2016?!

[sstarcher]

@jcmartins I'm not referring to metric related things. I'm referring to using Sensu for say http checks, and non-metric related activities.

[jcmartins]

do you know https://github.com/influxdb/telegraf/tree/master/plugins ?
And if this feature not exist you can write a plugin to it like in the sensu.

[m4ce]

+1 for this feature

[anlutro]

But with telegraf in all clients sending data to Influxdb and make alerts/checks in Kapacitor, you don't need use sensu anymore. ;-)

This simply isn't true, as InfluxDB + Kapacitor has no good way of handling "boolean" checks as of today - i.e. alerts for things like "sshd service is not running", "cronjob X has been running for more than an hour" etc.

I second this feature in Kapacitor, because I like it better than Sensu for sending alerts based on CPU/memory/disk/network usage, but Sensu fills the gaps, and I love having all events/notifications in one place (the Sensu/Uchiwa dashboard).

[nathanielc]

I agree Sensu integration would be a valuable addition to Kapacitor. Kapacitor is an alert source not an alert manager. Any takers on giving it a shot?

Here are very similar PRs for integrating with other tools #125 #139

[sstarcher]

@anlutro I fully agree Kapacitor solves a lot of problems, but it is not a generic solution like Sensu is and it would be silly to attempt to reimplement sensu features that are not metric related.

[anlutro]

@sstarcher I agree, I did not intend to make any argument for Kapacitor to become a more generic tool.

[anlutro]

It looks like you can't simply HTTP POST an event to Sensu, you have to do it through RabbitMQ. I guess this makes it somewhat more of an effort than previous output plugins...

[sstarcher]

@anlutro if you do a HTTP POST to a sensu client it will forward the data over to RabbitMQ for you.

[anlutro]

@sstarcher at least in my case, it's unreasonable to assume that I can HTTP POST to a sensu client from whatever server is running kapacitor.

[sstarcher]

@anlutro this resolves my use case and is the prefered method of Kapacitor communication with Sensu. If that's not workable for you I recommend you add support for RabbitMQ.

[m4ce]

@anlutro, you could run a sensu-client where kapacitor runs. The sensu-client would then listen on localhost:3030. I think that is fair enough.