How to manually prune oldest logstash indices

[jjbuchan]

When any of the log* nodes report excessive disk usage and a sudo du -kx --max-depth=3 / | sort -rn | head -10 shows the culprit is under /var/lib/elasticsearch, then the typical solution is to manually prune off the oldest logstash indices (created by day) to alleviate disk space.

On the node being fixed, first obtain the list of indices in basically date sorted order:

curl http://localhost:9200/_cat/indices|sort -k3

Looking at the third column, start working down the list of "logstash-*" indices performing an index delete operation until enough space is recovered (using df)

An example of a deletion is

curl -XDELETE http://localhost:9200/logstash-2016.09.16