When building complex, real-world Logstash filters, there can be a fair bit of processing logic. There are typically multiple grok patterns as well as fields used as flags for conditional processing.
The problem is, these intermediate extracted fields and processing flags are often ephemeral and unnecessary in your ultimate persistent store (e.g. ElasticSearch), but they will be inserted as fields unless you explicitly remove them.
One strategy is to use a mutate at the very end and remove any extra fields. A cleaner strategy that we will describe here is to declare these variables as @metadata field so they are never even considered for persistence.
Continue reading “ELK: metadata fields in Logstash for grok and conditional processing”
Logstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file.
Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, an incomplete date format, write to a file, or even make a web service call.
Continue reading “ELK: Using Ruby in Logstash filters”
When building your logstash filter, you would often like to validate your assumptions on a large sampling of input events without sending all the output to ElasticSearch.
Using Logstash metrics and conditionals, we can easily show:
- How many input events were processed successfully
- How many input events had errors
- An error file containing each event that processed in error
This technique gives you the ability to track your success rate across a large input set, and then do a postmortem review of each event that failed.
I’ll walk you through a Logstash conf file that illustrates this concept.
Continue reading “Logstash: Using metrics to debug the filtering process”