Page 1 of 1

Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 9:33 am
by hillhealthcenter
I have had to restart elasticsearch several times over the past couple days. Has anyone else had this issue?

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 2:48 pm
by lgroschen
Did your ./fullinstall show any failures when installing dependencies or anything else?

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 2:51 pm
by eloyd
Also, what are the specs for the machine you're running it on? We had an issue with an underpowered server where the JVM would crash.

Luckily, Nagios can monitor the Nagios Log Server and restart the JVM... :-)

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 3:06 pm
by hillhealthcenter
I deployed NLS from the VM OVF file using the default settings. I have a 2 instance cluster monitoring 5 hosts. Here are the specs for each instance:

VMware vCenter 5.5
CentOS 6.5 (Final)
CPU: 1 vCPU 1513 MHz
Memory: 2048 MB

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 3:14 pm
by eloyd
2GB of memory might be very tight for a bare install. That's what we were running. We upped it to 3GB and things calmed down, but we eventually moved it to a non-virtual 4GB machine instead.

If you can try upping your memory at all, that may help. Of course, you might also want to look in the /var/log/elasticsearch and/or /var/log/logstash directories for appropriate log file entries to see what crashed.

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 3:22 pm
by hillhealthcenter
Thanks, Eric!

I'll double the memory and add 3 more cores. I'll check the logs, too. :D

Much appreciated.

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 4:04 pm
by hillhealthcenter
The logs reveal that Elasticsearch crashed for lack of memory. The disk space is almost all in use, too. I'll add more free disk space to each instance.

What are steps to resize the partition after I allocate more disk space to each instance?

Re: Elasticsearch Database Offline

Posted: Mon Nov 03, 2014 5:35 pm
by slansing
You can actually follow the same guidelines here:

http://assets.nagios.com/downloads/nagi ... M_Disk.pdf

Re: Elasticsearch Database Offline

Posted: Tue Nov 04, 2014 3:02 pm
by hillhealthcenter
I got pvresize command not found error midway through the resize process. I decided to discard both instance and spin-up 2 new instances.

I appreciate the help. You guys are great. I'll let you know how it goes.