If your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths. This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation. The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health. In this article, I will … ELK: Deleting unassigned shards to restore cluster health
ElasticSearch very often serves as a repository for monitoring, logging, and business data. As such, integrations with external system are a requirement. The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server. We will use the olivere/elastic package for this purpose, it is … ELK: Connecting to ElasticSearch with a Go client
It is very common to have Logstash create time-based indexes in ElasticSearch that fit the format, <indexName>-YYYY.MM.DD. This means events submitted with @timestamp for that day all go to the same index. However, if you do not explicitly specify an index template that maps each field to a type, you can end up with unexpected query … ELK: Custom template mappings to force field types
ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch. In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script. One of the challenges of getting ElastAlert to run as a … ELK: Running ElastAlert as a service on Ubuntu 14.04
ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host. That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language. In this article we will describe installing … ELK: Installing MetricBeat for collecting system and application metrics
ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert. ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job
The Curator product from ElasticSearch allows you to apply batch actions to your indexes (close, create, delete, etc.). One specific use case is applying a retention policy to your indexes, deleting any indexes that are older than a certain threshold. Installation Start by installing Curator using apt and pip: $ sudo apt-get install python-pip -y … ELK: Using Curator to manage the size and persistence of your index storage
The ElasticSearch stack (ELK) is popular open-source solution that serves as both repository and search interface for a wide range of applications including: log aggregation and analysis, analytics store, search engine, and document processing. Its standard web front-end, Kibana, is a great product for data exploration and dashboards. However, if you have multiple data sources … Grafana: Connecting to an ElasticSearch datasource
The ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability. Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, … ELK: Architectural points of extension and scalability for the ELK stack
The heart of the ELK stack is Elasticsearch. In order to provide high availability and scalability, it needs to be deployed as a cluster with master and data nodes. The Elasticsearch cluster is responsible for both indexing incoming data as well as searches against that indexed data. Resources As described in the documentation, if there … ELK: Scaling an ElasticSearch Cluster