Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 6.x on Ubuntu 16.04.
Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example. This article is Part 4 of a series on Cloud Foundry concepts: Deploying the spring-music webapp, Part 1 Persisting spring-music data … CloudFoundry: Logging for the spring-music webapp, Part 4
Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. It supports multiple infrastructure platforms (EC2, VMware, OpenStack), and is able to standardize deployment, logging, scaling, and routing in a way that is friendly to a continuous delivery pipeline. In this series of articles, we will use the spring-music web application … CloudFoundry: Exploring Cloud Foundry using the spring-music application
ElasticSearch very often serves as a repository for monitoring, logging, and business data. As such, integrations with external system are a requirement. The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server. We will use the olivere/elastic package for this purpose, it is … ELK: Connecting to ElasticSearch with a Go client
It is very common to have Logstash create time-based indexes in ElasticSearch that fit the format, <indexName>-YYYY.MM.DD. This means events submitted with @timestamp for that day all go to the same index. However, if you do not explicitly specify an index template that maps each field to a type, you can end up with unexpected query … ELK: Custom template mappings to force field types
Logstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch. Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.
Logstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file. Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, … ELK: Using Ruby in Logstash filters
ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch. In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script. One of the challenges of getting ElastAlert to run as a … ELK: Running ElastAlert as a service on Ubuntu 14.04
ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host. That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language. In this article we will describe installing … ELK: Installing MetricBeat for collecting system and application metrics
ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job
The ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability. Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, … ELK: Architectural points of extension and scalability for the ELK stack
The heart of the ELK stack is Elasticsearch. In order to provide high availability and scalability, it needs to be deployed as a cluster with master and data nodes. The Elasticsearch cluster is responsible for both indexing incoming data as well as searches against that indexed data. Resources As described in the documentation, if there … ELK: Scaling an ElasticSearch Cluster
The most varied point in an ELK (Elasticsearch-Logstash-Kibana) stack is the mechanism by which custom events and logs will get sent to Logstash for processing. Companies running Java applications with logging sent to log4j or SLF4J/Logback will have local log files that need to be tailed. Applications running in containers may send everything to stdout/stderr, … ELK: Feeding the logging pipeline
Although the ELK stack has rich support for clustering, clustering is not supported over WAN connections due to Elasticsearch being sensitive to latency. There are also practical concerns of network throughput given how much data some installations index on an hourly basis. So as nice as it would be to have a unified, eventually consistent … ELK: Federated Search with a Tribe node
Kibana is the end user web application that allows us to query Elasticsearch data and create dashboards that can be used for analysis and decision making. Although Kibana can be pointed to any of the nodes in your Elasticsearch cluster, the best way to distribute requests across the nodes is to use a non-master, non-data … ELK: Pointing Kibana to a Client Node
The Logstash Indexing layer receives data from any number of input sources, transforms the data, and then submits it to Elasticsearch for indexing. Transforming and extracting data from every event can be both I/O as well as CPU intensive. Horizontal or Vertical Vertical scaling will only go so far in the Logstash indexing layer. In … ELK: Performance of the Logstash Indexing layer
Logging has always been a critical part of application development. But the rise of OS virtualization, applications containers, and cloud-scale logging solutions has turned logging into something bigger that managing local debug files. Modern applications and services are now expected to feed log aggregation and analysis stacks (ELK, Graylog, Loggly, Splunk, etc). This can be … Syslog: Sending Java log4j2 to rsyslog on Ubuntu