Stuck Initializing Shards count stuck at 1

[kmartin]

Hello,

From time to time I see a condition where our Nagios LS has Initializing Shards count stuck at 1. We've been able to determine the stuck shard is associated with the current day's index. Eventually it clears up and status goes to Green on its own (either the following day or once the index is closed on the 3rd day). My question is while this condition exists, what functionality, if any, is lost? In case it makes a difference this is a single node but we followed the guide for single instance deployment.

Thanks,
Ken

[cdienger]

Shards are where the data is stored so I would expect if one of them isn't available you'd be unable to any data it may contain so you may not be getting the full results when you do a search.

Is the status RED during this state? Check the logs in /var/log/elasticsearch/ for clues as to what the issue may be.

[Box293]

Can you check these guides and see if anything applies to your setup:

https://support.nagios.com/kb/article/n ... h-469.html
https://support.nagios.com/kb/article/n ... th-90.html

[kmartin]

Thanks Box293. I used those guides to determine the initializing shard was related to the current and active index.

Thanks cdienger. I searched /var/logs/elasticsearch and seems we run out of disk space overnight occasionally. Although we typically have over 150GB free of a 500GB disk. Also, it looks like RAM is tight and we should increase that -
Memory (Used/Free): 15.34 GB / 0 bytes
Swap (Used/Free): 701.50 MB / 0 bytes

I appreciate your help.

[cdienger]

One thing to keep mind with Elasticsearch is that it will allocate half of a systems total memory to itself. So of that 16GB, it will only use 8GB. Don't go over 64GB total on the system though - it can start having performance issues otherwise.

https://support.nagios.com/kb/article/n ... h-469.html has steps for increasing the low watermark on the system. This value that it is set to by default is 85%. You can increase the percentage or give it a hard value like 450GB. See https://www.elastic.co/guide/en/elastic ... cator.html for more examples.

[kmartin]

Is there more guidance on determining if RAM increase is in order? We consistently see 0 bytes under instance status -
Memory (Used/Free): 14.78 GB / 0 bytes
Swap (Used/Free): 771.11 MB / 0 bytes

We have 16G and as you point out, about half is dedicated to JVM -
Heap Committed: 7.72 GB

Is this one of those things which will use up all the RAM regardless of its size?

[cdienger]

Keep an eye on the heap usage under Admin > System > Instance Status. This should fluctuate as Elasticsearch uses it and java garbage collection clears things up. If it stays consistently high though upping the memory may be in order.