Elasticsearch Database Offline

[hillhealthcenter]

I have had to restart elasticsearch several times over the past couple days. Has anyone else had this issue?

[lgroschen]

Did your ./fullinstall show any failures when installing dependencies or anything else?

[eloyd]

Also, what are the specs for the machine you're running it on? We had an issue with an underpowered server where the JVM would crash.

Luckily, Nagios can monitor the Nagios Log Server and restart the JVM...

[hillhealthcenter]

I deployed NLS from the VM OVF file using the default settings. I have a 2 instance cluster monitoring 5 hosts. Here are the specs for each instance:

VMware vCenter 5.5
CentOS 6.5 (Final)
CPU: 1 vCPU 1513 MHz
Memory: 2048 MB

[eloyd]

2GB of memory might be very tight for a bare install. That's what we were running. We upped it to 3GB and things calmed down, but we eventually moved it to a non-virtual 4GB machine instead.

If you can try upping your memory at all, that may help. Of course, you might also want to look in the /var/log/elasticsearch and/or /var/log/logstash directories for appropriate log file entries to see what crashed.

[hillhealthcenter]

Thanks, Eric!

I'll double the memory and add 3 more cores. I'll check the logs, too.

Much appreciated.

[hillhealthcenter]

The logs reveal that Elasticsearch crashed for lack of memory. The disk space is almost all in use, too. I'll add more free disk space to each instance.

What are steps to resize the partition after I allocate more disk space to each instance?

[slansing]

You can actually follow the same guidelines here:

http://assets.nagios.com/downloads/nagi ... M_Disk.pdf

[hillhealthcenter]

I got pvresize command not found error midway through the resize process. I decided to discard both instance and spin-up 2 new instances.

I appreciate the help. You guys are great. I'll let you know how it goes.