The dig utility is convenient for doing manual DNS resolution from your system. Additionally, it uses the same OS resolver libraries as your applications which makes it more accurate than nslookup for emulating application issues and its output is more suitable for machine parsing. # ensure ‘dig’ is installed sudo apt install -y bind9-dnsutils dig … Bash: using dig for reverse DNS lookup by IP
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn
If your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths. This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation. The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health. In this article, I will … ELK: Deleting unassigned shards to restore cluster health
Cloud Foundry deployed Java applications can send log events to stdout/stderr and then a bound syslog drain can send this to a logging solution like ELK for ingestion. One area that has always been tricky when dealing with logging is multi-line Java stack traces. By default, because each line in the stack trace has a … CloudFoundry: Extracting Java multiline exception stack traces from Logback and Log4j2 using Logstash
Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 6.x on Ubuntu 16.04.
The two most common logging implementations used in conjunction with Spring/Spring Boot are Logback and Log4j2. In the recent past, a developer had a great deal of discretion on the format and files used for logging. But in the modern world of container deployment and scale, these logs typically feed enterprise logging solutions which requires … Java: Collapsing multiline stack traces into a single log event using Spring backed by Logback or Log4j2
Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example. This article is Part 4 of a series on Cloud Foundry concepts: Deploying the spring-music webapp, Part 1 Persisting spring-music data … CloudFoundry: Logging for the spring-music webapp, Part 4
Syslog is a message logging standard has been around for decades, but has renewed popularity as a method of log capture with the advent of containerization and centralized logging solutions. Enabling an Ubutu 14.04 or 16.04 host to act as a syslog server only takes a few simple steps.
ElasticSearch very often serves as a repository for monitoring, logging, and business data. As such, integrations with external system are a requirement. The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server. We will use the olivere/elastic package for this purpose, it is … ELK: Connecting to ElasticSearch with a Go client
It is very common to have Logstash create time-based indexes in ElasticSearch that fit the format, <indexName>-YYYY.MM.DD. This means events submitted with @timestamp for that day all go to the same index. However, if you do not explicitly specify an index template that maps each field to a type, you can end up with unexpected query … ELK: Custom template mappings to force field types
When building complex, real-world Logstash filters, there can be a fair bit of processing logic. There are typically multiple grok patterns as well as fields used as flags for conditional processing. The problem is, these intermediate extracted fields and processing flags are often ephemeral and unnecessary in your ultimate persistent store (e.g. ElasticSearch), but they will be inserted … ELK: metadata fields in Logstash for grok and conditional processing
Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.
Logstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file. Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, … ELK: Using Ruby in Logstash filters
Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events. This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers. If you do … Docker: logspout for Docker log collection
Log rotation is an essential maintenance task for managed servers. The logrotate package available in the main Ubuntu repository is easily configurable and is invoked by the cron service for automated log retention.
ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job
The Spring framework provides a proven and well documented model for the development of custom projects and services. The Spring Boot project takes an opinionated view of building production Spring applications, which favors convention over configuration. In this article we will explore how to configure a Spring Boot project to use the Simple Logging Facade … Spring: Spring Boot with SLF4J/Logback sending to syslog
In my previous posts, I have shown how to test grok patterns locally using Ruby on Linux and Windows. This works well when your VM do not have full internet access, or only have console access, or any reason that you want to test it locally. If you have access to a graphical web browser … Logstash: Testing Logstash grok patterns online
It is time consuming to restart the entire Logstash service and refeed it input when working on a grokparsefailure. Here is an easy way to test a line of input or log file against a grok pattern: