ELK: Using Ruby in Logstash filters

elastic-logstash-fwLogstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file.

Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, an incomplete date format, write to a file, or even make a web service call.

Continue reading “ELK: Using Ruby in Logstash filters”

Logstash: Using metrics to debug the filtering process

elastic-logstash-fw When building your logstash filter, you would often like to validate your assumptions on a large sampling of input events without sending all the output to ElasticSearch.

Using Logstash metrics and conditionals, we can easily show:

  • How many input events were processed successfully
  • How many input events had errors
  • An error file containing each event that processed in error

This technique gives you the ability to track your success rate across a large input set, and then do a postmortem review of each event that failed.

I’ll walk you through a Logstash conf file that illustrates this concept.

Continue reading “Logstash: Using metrics to debug the filtering process”