Logging

Python: New Relic instrumentation for Flask app deployed with Gunicorn

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production.  If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of Python: New Relic instrumentation for Flask app deployed with Gunicorn

ELK: Deleting unassigned shards to restore cluster health

If your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths. This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation.  The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health. In this article, I will ELK: Deleting unassigned shards to restore cluster health

CloudFoundry: Extracting Java multiline exception stack traces from Logback and Log4j2 using Logstash

Cloud Foundry deployed Java applications can send log events to stdout/stderr and then a bound syslog drain can send this to a logging solution like ELK for ingestion. One area that has always been tricky when dealing with logging is multi-line Java stack traces.  By default, because each line in the stack trace has a CloudFoundry: Extracting Java multiline exception stack traces from Logback and Log4j2 using Logstash

ELK: Installing Logstash on Ubuntu 16.04

Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 6.x on Ubuntu 16.04.

Java: Collapsing multiline stack traces into a single log event using Spring backed by Logback or Log4j2

The two most common logging implementations used in conjunction with Spring/Spring Boot are Logback and Log4j2. In the recent past, a developer had a great deal of discretion on the format and files used for logging.  But in the modern world of container deployment and scale, these logs typically feed enterprise logging solutions which requires Java: Collapsing multiline stack traces into a single log event using Spring backed by Logback or Log4j2

CloudFoundry: Logging for the spring-music webapp, Part 4

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example. This article is Part 4 of  a series on Cloud Foundry concepts: Deploying the spring-music webapp, Part 1 Persisting spring-music data CloudFoundry: Logging for the spring-music webapp, Part 4

ELK: Connecting to ElasticSearch with a Go client

ElasticSearch very often serves as a repository for monitoring, logging, and business data.  As such, integrations with external system are a requirement. The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server. We will use the olivere/elastic package for this purpose, it is ELK: Connecting to ElasticSearch with a Go client

ELK: Installing Logstash on Ubuntu 14.04

Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.

ELK: Using Ruby in Logstash filters

Logstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file. Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, ELK: Using Ruby in Logstash filters

Docker: logspout for Docker log collection

Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events. This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers. If you do Docker: logspout for Docker log collection

Ubuntu: logrotate for retention policy of logs

Log rotation is an essential maintenance task for managed servers.  The logrotate package available in the main Ubuntu repository is easily configurable and is invoked by the cron service for automated log retention.

ELK: ElastAlert for alerting based on data from ElasticSearch

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

ELK: ElasticDump and Python to create a data warehouse job

By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting ELK: ElasticDump and Python to create a data warehouse job

Spring: Spring Boot with SLF4J/Logback sending to syslog

The Spring framework provides a proven and well documented model for the development of custom projects and services. The Spring Boot project takes an opinionated view of building production Spring applications, which favors convention over configuration. In this article we will explore how to configure a Spring Boot project to use the Simple Logging Facade Spring: Spring Boot with SLF4J/Logback sending to syslog