ELK: Deleting unassigned shards to restore cluster health

elasticsearch-logoIf your ElasticSearch cluster is not healthy because of unassigned shards, there are multiple resolution paths.

This datadoghq article provides an excellent walk-through of how you can analyze and resolve the situation.  The simplest case is when those unassigned shards are not required anymore, and deleting them restores cluster health.

In this article, I will show how to do a simple listing of the unassigned shards, and the batch command that can delete them.

Before we move on, let me be clear that deleting the unassigned shard is not always the measure that should be taken.  You may want to reassign the shard instead.  But in this article, I am going to focus on a simple deletion.

First, we check on the cluster health and get the count of unassigned shards.

curl -XGET http://<elastichost>:9200/_cluster/health?pretty | grep unassigned_shards

The list of unassigned shards can be retrieved using:

curl -XGET http://<elastichost>:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'}

And if you want to delete every unassigned shard in the list above, you could send it to xargs and do a DELETE.

curl -XGET http://<elastichost>:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://<elastichost>:9200/{}"

 

REFERENCES

https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/ (various options for unassigned shards)

https://thoughts.t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode-ce196e20ba95

https://discuss.elastic.co/t/how-to-resolve-the-unassigned-shards/87635 (reroute to assign shard)

https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html (reroute command)

https://stackoverflow.com/questions/19967472/elasticsearch-unassigned-shards-how-to-fix (auto shard allocation, reassign shard manually, bash script to force reassignment)

https://www.cyberciti.biz/faq/linux-unix-bsd-xargs-construct-argument-lists-utility/ (xargs)

https://www.mkssoftware.com/docs/man1/xargs.1.asp (xargs -I {} equivalent to -i)