Monitoring

Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric


Categories: Kubernetes, Monitoring

HorizontalPodAutoscaler (HPA) allow you to dynamically scale the replica count of your Deployment based on basic CPU/memory resource metrics from the metrics-server.  If you want scaling based on more advanced scenarios and you are already using the Prometheus stack, the prometheus-adapter provides this enhancement. The prometheus-adapter takes basic Prometheus metrics, and then synthesizes custom API … Kubernetes: HorizontalPodAutoscaler evaluation based on Prometheus metric

Prometheus: installing kube-prometheus-stack on a kubeadm cluster


Categories: Kubernetes, Monitoring

The kube-prometheus-stack bundles the Prometheus Operator, monitors/rules, Grafana dashboards, and AlertManager needed to monitor a Kubernetes cluster. But there are customizations necessary to tailor the Helm installation for a Kubernetes cluster built using kubeadm. In this article, I will detail the necessary modifications to deploy a healthy monitoring stack on a kubeadm cluster.

Prometheus: monitoring services using additional scrape config for Prometheus Operator


Categories: Kubernetes, Monitoring

If you are running the Prometheus Operator (e.g. with kube-prometheus-stack) then you can specify additional scrape config jobs to monitor your custom services. An additional scrape config uses regex evaluation to find matching services en masse, and targets a set of services based on label, annotation, namespace, or name. Note that adding an additional scrape … Prometheus: monitoring services using additional scrape config for Prometheus Operator

Prometheus: monitoring a custom Service using ServiceMonitor and PrometheusRule


Categories: Kubernetes, Monitoring

If you are running the Prometheus Operator as part of your monitoring stack (e.g. kube-prometheus-stack) then you can have your custom Service monitored by defining a ServiceMonitor CRD. The ServiceMonitor is an object that defines the service endpoints that should be scraped by Prometheus and at what interval. In this article, we will deploy a … Prometheus: monitoring a custom Service using ServiceMonitor and PrometheusRule

Prometheus: adding a Grafana dashboard using a ConfigMap


Categories: Kubernetes, Monitoring

If your Grafana deployment is using a sidecar to watch for new dashboards defined as a ConfigMap, then adding a dashboard is a dynamic operation that can be done without even restarting the pod. If you have deployed the Prometheus/Grafana stack with kube-prometheus-stack, then you can check for the existence of the ‘grafana-sc-dashboard’ sidecar using: … Prometheus: adding a Grafana dashboard using a ConfigMap

Prometheus: external template for AlertManager html email with kube-prometheus-stack


Categories: Kubernetes, Monitoring

The kube-prometheus-stack bundles AlertManager for taking action on Prometheus alerts. And if you are customizing the Heml custom values file to configure email alerting, there are multiple options available.  The simplest is to allow the system to fallback to using the default subject and html templates. But if you need to tailor the email content … Prometheus: external template for AlertManager html email with kube-prometheus-stack

Prometheus: exposing Prometheus/Grafana as Ingress for kube-prometheus-stack


Categories: Kubernetes, Monitoring

The kube-prometheus-stack bundles Prometheus, Grafana, and AlertManager for monitoring a Kubernetes cluster. By default, the Ingress of these services is disabled.  In this article I will show you how to expose these services with NGINX Ingress either via subdomain (e.g. prometheus.my.domain) or web context (e.g. my.domain/prometheus). You would not want to expose these monitoring applications … Prometheus: exposing Prometheus/Grafana as Ingress for kube-prometheus-stack

Prometheus: installing kube-prometheus-stack on K3s cluster


Categories: Kubernetes, Monitoring

The kube-prometheus-stack bundles the Prometheus Operator, monitors/rules, Grafana dashboards, and AlertManager needed to monitor a Kubernetes cluster. But there are customizations necessary to tailor the Helm installation for K3s, a lightweight Kubernetes installation. In this article, I will detail the necessary modifications to deploy a healthy monitoring stack on a K3s cluster.

Zabbix: Using Docker Compose to install and upgrade Zabbix


Categories: Containers, Monitoring

Zabbix distributes Docker images for each component.  Not only does this mean you can quickly standup the monitoring solution, but upgrades also become a simple matter of trading up images. In this article, I will show how to stand up and then upgrade a zabbix installation using docker-compose.

CloudFoundry: Monitoring the spring-music webapp, Part 5


Categories: Containers, Java, Monitoring

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example. This article is Part 5 of  a series on Cloud Foundry concepts: Deploying the spring-music webapp, Part 1 Persisting spring-music data … CloudFoundry: Monitoring the spring-music webapp, Part 5

Zabbix: LLD low-level discovery returning multiple values


Categories: Monitoring, Scripting

Zabbix low-level discovery (LLD) provides a way to create an array of related items, triggers, or graphs without needing to know the exact number of entities up front. The easiest way to populate the keys of a discovery item is to add a “UserParameter” in zabbix_agentd.conf, and then the Zabbix agent will  invokes a script which returns the set … Zabbix: LLD low-level discovery returning multiple values

Zabbix: Accessing Zabbix using the py-zabbix Python module


Categories: Monitoring, Python

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.

ELK: Running ElastAlert as a service on Ubuntu 14.04


Categories: Linux, Monitoring

ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch. In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script. One of the challenges of getting ElastAlert to run as a … ELK: Running ElastAlert as a service on Ubuntu 14.04

ELK: Installing MetricBeat for collecting system and application metrics


Categories: DevOps, Monitoring

ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host.  That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language. In this article we will describe installing … ELK: Installing MetricBeat for collecting system and application metrics

ELK: ElastAlert for alerting based on data from ElasticSearch


Categories: DevOps, Linux, Logging, Monitoring

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

Docker: Sending Spring Boot logging to syslog


Categories: Containers, DevOps, Java, Monitoring

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging.  But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space. The … Docker: Sending Spring Boot logging to syslog