Question: how best to integrate with PagerDuty?

[sdanieru]

Greetings!
I'd like to integrate your icinga2 image w/ PagerDuty. What's the best way to do this? Per this page:
https://www.pagerduty.com/docs/guides/icinga2-integration-guide/
I've made a Dockerfile to wrap your image w/ the additional changes mentioned in the integration guide, but I'm having difficulty getting both the cron and /opt/run to run at the same time.
Here's what I have so far:

FROM jordan/icinga2:latest

RUN apt-get update \
  && apt-get install -y libwww-perl libcrypt-ssleay-perl libsys-syslog-perl cron

COPY --chown=nagios:root conf/pagerduty-icinga2.conf /etc/icinga2/conf.d/pagerduty-icinga2.conf
COPY --chown=nagios:root conf/templates.conf /etc/icinga2/conf.d/templates.conf
COPY --chown=nagios:root bin/pagerduty_icinga.pl /usr/local/bin/pagerduty_icinga.pl
COPY hello-cron /etc/cron.d/hello-cron

CMD ["&& cron -f"]
-

since ENTRYPOINT is used in the base img, CMD ought to append '&& cron' - but it looks like cron hasn't started, as it's not writing to the log. As I'm thinking about this more, perhaps it's never getting to '&& cron' since /opt/run is still running...

Not sure if I'm on the right track here, or if there's a better way. Any help appreciated!

[jjethwa]

Hi @sdanieru

I would probably use supervisord (it's already installed and running icinga, mysql, etc) to run the task you need or to run the cron daemon instead. Hope that helps :)

[sdanieru]

that is helpful, thanks for the quick response, appreciate it! If I can get it working I'll leave details here.

[jjethwa]

Sounds good, thanks @sdanieru Let me know if you need any help 😄

[sdanieru]

so I've successfully wrapped your image w/ the specifics that the pagerduty integration needed. When I started the container it had some built-in checks that immediately went critical (e.g. icinga2:apt) and it did send the alerts to pagerduty, so it does work, but I am somewhat confused, as the pre-installed service checks don't show up in 'Icinga Director' > Services and it's unclear to me how to 'hook' into the pagerduty notifications.
The services I've setup, I see them correctly going from OK to Critical when I bring the monitored service down, but they never send notifications. In the service editor, there's no way to add a notification.
One approach I tried was to add the service to a group named 'prod' and then in the notification under 'assign where' I added
["service.groups"] contains prod
but this didn't work.
Another item I noticed, is in the built-in alerts, they list several contacts and contactGroup, which included the PagerDuty Notifier, but I don't see these contacts or contactGroup anywhere in the Icinga Director.

[jjethwa]

Hi @sdanieru

That's great news about the integration. I'm not a Director expert , but I believe that it does not show objects that are pre-defined (/etc/icinga2/conf.d/ or /usr/share/icinga2). Just to make sure, you set up:

1. Notification template
2. NotificationCommand
3. Notification
4. UserGroup
5. User

That's the minimum setup needed IMO. Take a look at my Slack notification repo: https://github.com/jjethwa/icinga2-slack-notification for an example