ELK: Running ElastAlert as a service on Ubuntu 14.04

ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch.

In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script.

One of the challenges of getting ElastAlert to run as a service is that is has a very strict set of module requirements that very easily conflicts with other Python applications, and so we will use Python’s virtualenv to build it in isolation and then call that wrapper from the service script.

Installation

First, let’s install the ElastAlert:

$ python --version

$ cd /tmp

$ sudo apt-get install git software-properties-common python python-pip -y

$ sudo apt-get install python-dev libffi-dev libssl-dev -y

$ sudo pip install virtualenv

$ sudo pip install "setuptools>=11.3"

$ git clone https://github.com/Yelp/elastalert.git

$ virtualenv elastalert

$ cd elastalert

$ sudo sh -c ". bin/activate; python setup.py install"

$ sudo -H sh -c ". bin/activate; pip install 'elasticsearch<3.0.0'"

If you see traceback errors at the very end of the pip installations, you can typically ignore them, they are non-fatal.

Now you should be able to run elastalert using the virtualenv wrapper, with the expected error message that there is “No such file or directory: ‘config.yaml'”.

$ bin/elastalert --verbose

If you had problems with the install or requirements, you would see a different error message related to modules or dependencies.

Configuration

The main configuration is done in config.yaml.

$ cp config.yaml.example config.yaml

$ vi config.yaml

At the minimum, make sure to change the ‘es_host’ key to point to your ElasticSearch server, but we will also have ElastAlert check for our rule conditions every 10 seconds for debugging purposes.

run_every:
  seconds: 10
es_host: esmaster
es_port: 9200

Notice that by default, ElastAlert will be executing against all the rules in ‘rules_folder: examples_rules’.

ElasticSearch Index Creation

ElastAlert saves information about its queries/alerts back to an ES index named ‘elastalert_status’, create this index using the following commands. Press <ENTER> twice to accept the default index name and question asking about name of existing index.

$ bin/elastalert-create-index

Validate from Console

If unchanged, the config.yaml will run against all the rules in the folder ‘example_rules’.

$ bin/elastalert --config config.yaml --verbose

Running as a Service

Now it’s time to run ElastAlert as a service, we will first grab the System-V init script from my github project.

$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/elastalert -O /etc/init.d/elastalert

$ sudo chmod 755 /etc/init.d/elastalert

$ sudo update-rc.d elastalert defaults 95 10

$ sudo vim /etc/init.d/elastalert

Then modify ‘/etc/init.d/elastalert’, EA_DIR is currently set to ‘/tmp/elastalert’ as described in this article, but clearly that is not where you want it in production. And RULE_OPTS can be modified if you only want to evaluate a single rule rather than the ‘rules_folder’ value of the main config.

Next, add an ‘elastalert’ user which is used for process/log ownership and start the service.

$ sudo useradd elastalert

$ sudo service elastalert status

$ sudo service elastalert start

The process ID and logs can be found using the following commands:

cat /var/run/elastalert/elastalert.pid

tail -f /var/log/elastalert/elastalert.log

Custom Rules

If you want to explore creating custom rules, such as triggers on high CPU load as reported by a Metricbeat agent to ElasticSearch, read my first article that describes how to collect and then create a custom rule.

REFERENCES

https://github.com/Yelp/elastalert/issues/194

http://python-guide-pt-br.readthedocs.io/en/latest/dev/virtualenvs/

https://medium.com/the-python-corner/using-virtual-environments-with-python-7166d3bfa218

https://github.com/slackhq/go-audit/tree/master/examples/elastalert

https://askubuntu.com/questions/234758/how-to-use-a-python-virtualenv-with-sudo

https://unix.stackexchange.com/questions/68753/starting-server-with-nohup-and-redirecting-input-output

http://www.linuxhowtos.org/manpages/8/start-stop-daemon.htm

http://archives.aidanfindlater.com/blog/2009/09/04/sample-init-d-script/

http://stackoverflow.com/questions/29134512/insecureplatformwarning-a-true-sslcontext-object-is-not-available-this-prevent

https://github.com/fabianlee/blogcode/blob/master/elastalert

ElastAlert service with SystemD