Nagios Log server Elasticsearch issue

[SrinivasanNSL]

Hi Team,

at recent times i am facing issues in elasticsearch service in nagios log server. At times the service goes in to active(exited) mode when i check status of the service. Had checked couple of nagios forums and they are suggestig to modify the heap size of elasticsearch.

Resolution: When I reboot the server the service runs fine.

I have 2 vCPU and 8 GB of Memory configuration for my Nagios log server. Please suggest the recommended configurations for elasticsearch and logstash .

[scottwilkerson]

How much log data are you sending to Log Server?

Is this just a single instance?

Can you send the log in /var/log/elasticsearch/*.log

[SrinivasanNSL]

HI,

Yes its a single instance. PFA logs.

[scottwilkerson]

Hmm, in the logs you sent I see this line

Code: Select all

[2018-04-10 04:12:23,553][WARN ][common.network           ] failed to resolve local host, fallback to loopback

Somehow the machine wasn't able to resolve the localhost however 4 seconds later did recover

This was the only error shown in the weeks worth of logs. When did this issue describe last happen?

[SrinivasanNSL]

Scott,

The Issue happened on 10th April 2018. While accessing the Nagios Log server URL it says waiting for the Database to Startup. When I logged in through Putty session and check for elastic search Service status it says " active (Exited)". I did restart of the service but no luck. I did reboot of the server and the services are back to normal.

[scottwilkerson]

Hmm, I could not locate that anywhere in the logs. If it happens again you should sty simply restarting the elasticsearch service.

[SrinivasanNSL]

Thanks Scott.

But last time during this Issue the Service restart doesnt work.

Also can you please let me know what could be the ideal configuration of heap options in elasticsearch file for my VM configuration?

Will the above option will help in resolving this issue?

[scottwilkerson]

Log Server dynamically assign heap to be about 1/2 total memory. This is the ideal amount, until you have over 64GB of RAM, then the ideal amount is exactly 32GB.

If you have heavily leaded machine to fix a heap exhausted error you would need to add more total memory to each instance in your cluster

[SrinivasanNSL]

ok Scott.

I am preparing a problem statement to share with my management. Can you please let me know what could be the possible causes for this kind of behaviour? and any resolution if its there.

I will update accordingly.

[scottwilkerson]

ok Scott.

I am preparing a problem statement to share with my management. Can you please let me know what could be the possible causes for this kind of behaviour? and any resolution if its there.

I will update accordingly.

It is very likely if you saw Heap Error messages you need more RAM in each of your instances. Adding RAM should resolve the issue.