Zabbix: LLD low-level discovery returning multiple values

Zabbix low-level discovery (LLD) provides a way to create an array of related items, triggers, or graphs without needing to know the exact number of entities up front.

The easiest way to populate the keys of a discovery item is to add a “UserParameter” in zabbix_agentd.conf, and then the Zabbix agent will  invokes a script which returns the set of keys.

But the keys are only the first part of a real solution, because what you really want to send back are the values associated with those keys.  For example, if you are monitoring a database, you don’t want to just send the list of tables available, you may want to send back each table name and then its row count and size on disk.

Unfortunately Zabbix does not support sending back multiple values [1,2,3,4].  There are various workarounds such as using one UserParameter for the discovery key and another with a UserParameter=key[*] to fetch each row of data, or using vfs.file.regexp to parse values that have been written to a file.

But I think the cleanest solution, and one that requires the minimal number of spawned processes on the agent host is to invoke zabbix_sender from inside the script to send back all the values you want to populate.

Continue reading “Zabbix: LLD low-level discovery returning multiple values”

Zabbix: Sending Zabbix metrics using a Go client

The open-source Zabbix monitoring solution has a published, simple binary protocol that allows you to send metrics to the Zabbix server without relying on the Zabbix Agent – which makes it very convenient for integration with other parts of your infrastructure.

In this article, I’ll show how to use the go-zabbix package for sending metrics to the Zabbix server.  If instead you were looking to manipulate the backend server definitions (host, templates, hostgroups, etc.) using the REST API, then see my other article here.

Continue reading “Zabbix: Sending Zabbix metrics using a Go client”

Zabbix: Zabbix REST API using a Go client

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems.

This fosters development of community-driven modules like Ryan Day’s zabbix Go language package, which is an easy way to automate Zabbix tasks like creating hosts and manipulating other back end structures.

One of the nice things about the Go language is that libraries are generally statically linked into a single executable, so you only need to copy over a single executable.  You don’t have to copy 3rd party jars (Java) or require an internet connection to pypi from production system (Python).

Continue reading “Zabbix: Zabbix REST API using a Go client”

Zabbix: Accessing Zabbix using the py-zabbix Python module

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems.

This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.

Continue reading “Zabbix: Accessing Zabbix using the py-zabbix Python module”

Zabbix: Installing a Zabbix Agent on Ubuntu 14.04

The open-source Zabbix monitoring solution has very lightweight agents that are easy to install on Ubuntu.

Although the Ubuntu main repository has a build available, it is older and so we are going to choose to download and install the latest point version in this article.  Unfortunately, the repo.zabbix.com cannot be added directly as an Ubuntu repository source.

Continue reading “Zabbix: Installing a Zabbix Agent on Ubuntu 14.04”

Grafana: Connecting to a Zabbix datasource

Zabbix is an open-source monitoring solution that provides metrics collection, dynamic indexes, alerting, dashboards, and an API for external integration.  But graphing is arguably one Zabbix’s weak points; it still builds static images while other enterprise and consumer applications have set end users’ expectations for graph visualization and interactivity very high.

Luckily, the Zabbix plugin for Grafana can put a facelift on the valuable data stored in Zabbix.  With this new data source, your end users can get the beautiful dashboard view they expect from a modern application.

Continue reading “Grafana: Connecting to a Zabbix datasource”

Zabbix: Enabling API fetch of Trend data in Zabbix2

Until Zabbix3, trend data was not available via the Zabbix API.  This meant that you could retrieve the  raw values of a key over time, but not the aggregated historical trends of that value (e.g. CPU average over 5 minute intervals).

The only way to monitor trends was to look at the visual graph generated by Zabbix or query the underlying database directly.  Meanwhile, graphs are arguably one of Zabbix’s weak points, especially given newer solutions like Grafana.

This was a major oversight in Zabbix2 functionality, and led to community patches that enabled this functionality in Zabbix 2.x.  With this trend data now exposed, the community was free to write custom alerting, graphing, and capacity planning tools.  For example, the Zabbix plugin for Grafana relies on this patch when the data source is Zabbix 2.x.

Continue reading “Zabbix: Enabling API fetch of Trend data in Zabbix2”

Zabbix: Alert to PagerDuty using Zabbix3

Having Zabbix send alert mails directly to user groups is typically outgrown as the system matures and the number of alerts increase, new lines of business and engineering groups are on-boarded, and on-call scheduling is implemented.

If you already use PagerDuty for on-call scheduling, then it makes perfect sense to have Zabbix create incidents in PagerDuty. While it is possible to use standard email to perform some level of integration,  the native library is the tightest integration you will find and supports multiple pager duty services.

The agent built by PagerDuty is especially well done, using their API to automatically create PagerDuty incidents as well as automatically mark them resolved if the trigger is only ephemeral (e.g. a temporary cpu spike).

Continue reading “Zabbix: Alert to PagerDuty using Zabbix3”