ELK: Architectural points of extension and scalability for the ELK stack

elasticsearch-logoThe ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability.

Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, message queues, indexes on distinct physical drives, etc… So in this article I want to explore the obvious extension points, and encourage the reader to treat this as a starting point in their own design and deployment.

Continue reading “ELK: Architectural points of extension and scalability for the ELK stack”

ELK: Performance of the Logstash Indexing layer

elasticsearch-logoThe Logstash Indexing layer receives data from any number of input sources, transforms the data, and then submits it to Elasticsearch for indexing.  Transforming and extracting data from every event can be both I/O as well as CPU intensive.

Horizontal or Vertical

Vertical scaling will only go so far in the Logstash indexing layer.  In order to keep up with the processing demand as well as provide availability, horizontal scalability must be employed.

And if you are going to have vertical scaling, you should be using either configuration management (SaltStack, Ansible, etc.) or containers to be able to create extra Logstash indexing instances without excessive manual steps.

Continue reading “ELK: Performance of the Logstash Indexing layer”