Page 1 of 2
Nagios Log server Elasticsearch issue
Posted: Fri Apr 13, 2018 7:52 am
by SrinivasanNSL
Hi Team,
at recent times i am facing issues in elasticsearch service in nagios log server. At times the service goes in to active(exited) mode when i check status of the service. Had checked couple of nagios forums and they are suggestig to modify the heap size of elasticsearch.
Resolution: When I reboot the server the service runs fine.
I have 2 vCPU and 8 GB of Memory configuration for my Nagios log server. Please suggest the recommended configurations for elasticsearch and logstash .
Re: Nagios Log server Elasticsearch issue
Posted: Fri Apr 13, 2018 9:22 am
by scottwilkerson
How much log data are you sending to Log Server?
Is this just a single instance?
Can you send the log in /var/log/elasticsearch/*.log
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 7:40 am
by SrinivasanNSL
HI,
Yes its a single instance. PFA logs.
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 7:49 am
by scottwilkerson
Hmm, in the logs you sent I see this line
Code: Select all
[2018-04-10 04:12:23,553][WARN ][common.network ] failed to resolve local host, fallback to loopback
Somehow the machine wasn't able to resolve the localhost however 4 seconds later did recover
This was the only error shown in the weeks worth of logs. When did this issue describe last happen?
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 8:03 am
by SrinivasanNSL
Scott,
The Issue happened on 10th April 2018. While accessing the Nagios Log server URL it says waiting for the Database to Startup. When I logged in through Putty session and check for elastic search Service status it says " active (Exited)". I did restart of the service but no luck. I did reboot of the server and the services are back to normal.
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 8:31 am
by scottwilkerson
Hmm, I could not locate that anywhere in the logs. If it happens again you should sty simply restarting the elasticsearch service.
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 8:49 am
by SrinivasanNSL
Thanks Scott.
But last time during this Issue the Service restart doesnt work.
Also can you please let me know what could be the ideal configuration of heap options in elasticsearch file for my VM configuration?
Will the above option will help in resolving this issue?
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 8:59 am
by scottwilkerson
Log Server dynamically assign heap to be about 1/2 total memory. This is the ideal amount, until you have over 64GB of RAM, then the ideal amount is exactly 32GB.
If you have heavily leaded machine to fix a heap exhausted error you would need to add more total memory to each instance in your cluster
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 9:29 am
by SrinivasanNSL
ok Scott.
I am preparing a problem statement to share with my management. Can you please let me know what could be the possible causes for this kind of behaviour? and any resolution if its there.
I will update accordingly.
Re: Nagios Log server Elasticsearch issue
Posted: Mon Apr 16, 2018 9:34 am
by scottwilkerson
SrinivasanNSL wrote:ok Scott.
I am preparing a problem statement to share with my management. Can you please let me know what could be the possible causes for this kind of behaviour? and any resolution if its there.
I will update accordingly.
It is very likely if you saw Heap Error messages you need more RAM in each of your instances. Adding RAM should resolve the issue.