In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script.
One of the challenges of getting ElastAlert to run as a service is that is has a very strict set of module requirements that very easily conflicts with other Python applications, and so we will use Python’s virtualenv to build it in isolation and then call that wrapper from the service script.
First, let’s install the ElastAlert:
$ python --version $ cd /tmp $ sudo apt-get install git software-properties-common python python-pip -y $ sudo apt-get install python-dev libffi-dev libssl-dev -y $ sudo pip install virtualenv $ sudo pip install "setuptools>=11.3" $ git clone https://github.com/Yelp/elastalert.git $ virtualenv elastalert $ cd elastalert $ sudo sh -c ". bin/activate; python setup.py install" $ sudo -H sh -c ". bin/activate; pip install 'elasticsearch<3.0.0'"
If you see traceback errors at the very end of the pip installations, you can typically ignore them, they are non-fatal.
Now you should be able to run elastalert using the virtualenv wrapper, with the expected error message that there is “No such file or directory: ‘config.yaml'”.
$ bin/elastalert --verbose
If you had problems with the install or requirements, you would see a different error message related to modules or dependencies.
The main configuration is done in config.yaml.
$ cp config.yaml.example config.yaml $ vi config.yaml
At the minimum, make sure to change the ‘es_host’ key to point to your ElasticSearch server, but we will also have ElastAlert check for our rule conditions every 10 seconds for debugging purposes.
run_every: seconds: 10 es_host: esmaster es_port: 9200
Notice that by default, ElastAlert will be executing against all the rules in ‘rules_folder: examples_rules’.
ElasticSearch Index Creation
ElastAlert saves information about its queries/alerts back to an ES index named ‘elastalert_status’, create this index using the following commands. Press <ENTER> twice to accept the default index name and question asking about name of existing index.
Validate from Console
If unchanged, the config.yaml will run against all the rules in the folder ‘example_rules’.
$ bin/elastalert --config config.yaml --verbose
Running as a Service
Now it’s time to run ElastAlert as a service, we will first grab the System-V init script from my github project.
$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/elastalert -O /etc/init.d/elastalert $ sudo chmod 755 /etc/init.d/elastalert $ sudo update-rc.d elastalert defaults 95 10 $ sudo vim /etc/init.d/elastalert
Then modify ‘/etc/init.d/elastalert’, EA_DIR is currently set to ‘/tmp/elastalert’ as described in this article, but clearly that is not where you want it in production. And RULE_OPTS can be modified if you only want to evaluate a single rule rather than the ‘rules_folder’ value of the main config.
Next, add an ‘elastalert’ user which is used for process/log ownership and start the service.
$ sudo useradd elastalert $ sudo service elastalert status $ sudo service elastalert start
The process ID and logs can be found using the following commands:
cat /var/run/elastalert/elastalert.pid tail -f /var/log/elastalert/elastalert.log
If you want to explore creating custom rules, such as triggers on high CPU load as reported by a Metricbeat agent to ElasticSearch, read my first article that describes how to collect and then create a custom rule.
ElastAlert service with SystemD
sudo vi lib/python2.7/site-packages/exotel-0.1.1-py2.7.egg-info/requires.txt (modify requirements manually)
sudo update-rc.d -f elastalert defaults 95 10 (force removal)
$ sudo -H sh -c “. bin/activate; pip install ‘stomp.py==4.1.17′” (not necessary before install)
$ sudo -H sh -c “. bin/activate; pip install requests[security]” (not necessary before install)
$ sudo -H sh -c “. bin/activate; pip install pyOpenSSL ndg-httpsclient pyasn1” (not necessary before older ES module installed)