Zabbix: Alert to PagerDuty using Zabbix3

Having Zabbix send alert mails directly to user groups is typically outgrown as the system matures and the number of alerts increase, new lines of business and engineering groups are on-boarded, and on-call scheduling is implemented.

If you already use PagerDuty for on-call scheduling, then it makes perfect sense to have Zabbix create incidents in PagerDuty. While it is possible to use standard email to perform some level of integration, the native library is the tightest integration you will find and supports multiple pager duty services.

The agent built by PagerDuty is especially well done, using their API to automatically create PagerDuty incidents as well as automatically mark them resolved if the trigger is only ephemeral (e.g. a temporary cpu spike).

Installation of the PagerDuty agent is done on the Zabbix server. There are OS level steps for agent installation/configuration, and then there are configuration changes made in the Zabbix Web GUI. There is no reason for me to lay out the full instructions, the guides for Zabbix3 and Zabbix2 integration at the PagerDuty site are very detailed. Their troubleshooting guide is helpful as well.

I would point out one warning in the installation steps, make sure you check the value of ‘AlertScriptsPath’ in your /etc/zabbix_server.conf. This needs to match the path where you create a link to the pdagent binary. On my Ubuntu 14.04 system using Zabbix 3.2, I set the value of AlertScriptsPath to ‘/usr/lib/zabbix/alertscripts’ and changed the ‘ln’ command accordingly.

As the agent installation instructions detail, be sure that sending messages using ‘pd-send’ works properly before attempting to move forward with Zabbix configuration changes.

If you want to monitor the trigger alerts being sent to PagerDuty, tail the pdagent log.