Syslog is a message logging standard has been around for decades, but has renewed popularity as a method of log capture with the advent of containerization and centralized logging solutions.
Enabling an Ubutu 14.04 or 16.04 host to act as a syslog server only takes a few simple steps.
Continue reading “Ubuntu: Enabling syslog on Ubuntu and custom templates”
ElasticSearch very often serves as a repository for monitoring, logging, and business data. As such, integrations with external system are a requirement.
The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server.
We will use the olivere/elastic package for this purpose, it is well maintained and has support for both ElasticSearch 5.x and 2.x depending on your import statement. In this article, we will be hitting an ElasticSearch 2.x backend.
Continue reading “ELK: Connecting to ElasticSearch with a Go client”
It is very common to have Logstash create time-based indexes in ElasticSearch that fit the format, <indexName>-YYYY.MM.DD. This means events submitted with @timestamp for that day all go to the same index.
However, if you do not explicitly specify an index template that maps each field to a type, you can end up with unexpected query results. The reason is that without explicit mappings, the index (that is created fresh each day) uses its best judgement to assign field types based on the first event inserted.
In this article, I’ll show you how to create explicit custom index templates so that field types are uniform across your time-series indexes.
Continue reading “ELK: Custom template mappings to force field types”
When building complex, real-world Logstash filters, there can be a fair bit of processing logic. There are typically multiple grok patterns as well as fields used as flags for conditional processing.
The problem is, these intermediate extracted fields and processing flags are often ephemeral and unnecessary in your ultimate persistent store (e.g. ElasticSearch), but they will be inserted as fields unless you explicitly remove them.
One strategy is to use a mutate at the very end and remove any extra fields. A cleaner strategy that we will describe here is to declare these variables as @metadata field so they are never even considered for persistence.
Continue reading “ELK: metadata fields in Logstash for grok and conditional processing”
Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.
Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.
Continue reading “ELK: Installing Logstash on Ubuntu 14.04”
Logstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file.
Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, an incomplete date format, write to a file, or even make a web service call.
Continue reading “ELK: Using Ruby in Logstash filters”
Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events.
This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers.
If you do not have docker installed yet, see my article here. Before moving on, you should be able to run the hello-world container.
Continue reading “Docker: logspout for Docker log collection”
Log rotation is an essential maintenance task for managed servers. The logrotate package available in the main Ubuntu repository is easily configurable and is invoked by the cron service for automated log retention.
Continue reading “Ubuntu: logrotate for retention policy of logs”
ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert.
ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.
Continue reading “ELK: ElastAlert for alerting based on data from ElasticSearch”
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized.
ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting the data as json for later import into a warehouse like Hadoop. ElasticDump does not have a special filter for time based indexes (index-YYYY.MM.DD), so you must specify exact index names.
In this article we will use Python to query a source ElasticSearch instance (an instance meant for near real-time querying, keeps minimal amount of data), and exports any indexes from the last 14 days into a target ElasticSearch instance (an instance meant for data warehousing, has more persistent storage and users expect multi-second query times).
Continue reading “ELK: ElasticDump and Python to create a data warehouse job”
The Spring framework provides a proven and well documented model for the development of custom projects and services. The Spring Boot project takes an opinionated view of building production Spring applications, which favors convention over configuration.
In this article we will explore how to configure a Spring Boot project to use the Simple Logging Facade for Java (SLF4J) with a Logback backend to send log events to the console, filesystem, and syslog.
Continue reading “Spring: Spring Boot with SLF4J/Logback sending to syslog”
In my previous posts, I have shown how to test grok patterns locally using Ruby on Linux and Windows. This works well when your VM do not have full internet access, or only have console access, or any reason that you want to test it locally.
If you have access to a graphical web browser and the log file, there is a nice online grok constructor here and here. and by simply entering a sampling of the log lines and a grok pattern, you can verify that all the lines are parsed correctly.
Here is a small example to start you off:
Continue reading “Logstash: Testing Logstash grok patterns online”
It is time consuming to restart the entire Logstash service and refeed it input when working on a grokparsefailure. Here is an easy way to test a line of input or log file against a grok pattern:
Continue reading “Logstash: Testing Logstash grok patterns locally on Linux”