The Curator product from ElasticSearch allows you to apply batch actions to your indexes (close, create, delete, etc.). One specific use case is applying a retention policy to your indexes, deleting any indexes that are older than a certain threshold.
Installation
Start by installing Curator using apt and pip:
$ sudo apt-get install python-pip -y $ sudo pip install elasticsearch-curator $ /usr/local/bin/curator --version
Configuration
Use a file named ‘curator-config.yml’ to configure the product:
client:
hosts:
- 127.0.0.1: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: True
http_auth:
timeout: 30
master_only: False
logging:
loglevel: INFO
logfile:
logformat: default
blacklist: ['elasticsearch', 'urllib3']
Action File
The next file, ‘curator-action.yml’ will define the list of actions we wish to perform against the ElasticSearch instance. In the file below we will close indexes after 14 days to save RAM, and delete indexes after 30 days to reduce the persistence storage needs.
actions:
1:
action: close
description: close indices
options:
delete_aliases: False
timeout_override:
continue_if_exception: True
disable_action: False
filters:
-
filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 14
exclude:
2:
action: delete_indices
description: delete indices
filters:
-
filtertype: age
source: name
direction: older
timestring: '%Y.%m.%d'
unit: days
unit_count: 30
exclude:
Running
The only thing left to do is run Curator with the main configuration and action file:
$ /usr/local/bin/curator --config curator-config.yml curator-action.yml
This action would typically be put into a nightly cron job
$ sudo service crontab status $ sudo crontab -l -u root $ sudo crontab -e
When prompted to edit the crontab list, you would use the line below, making sure to use fully qualified paths to the config and log files.
0 0 * * * /usr/local/bin/curator --config /tmp/curator-config.yml /tmp/curator-action.yml >> /tmp/curator.log 2>&1
This would run the Curator actions every day at midnight. For testing, you may want to run this every minute (replace ‘0 0’ with ‘* *’), and tail the curator.log to ensure it is running properly.
REFERENCES
https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/about.html