Page 1 of 1

Nagios Log Server is not collecting Logs from the machines

Posted: Mon Dec 03, 2018 3:40 am
by srinivasmandalika
Hello,

We are facing issue with our Nagios Log Server of not collecting logs from the servers...

First I see that Logstash process is dying frequently even after restart and followed procedure mentioned in https://support.nagios.com/kb/article/n ... g-576.html

Then, I see the home page states it receiving logs from only 1 host... I followed https://support.nagios.com/kb/article.php?id=38 ... But, No Luck

I followed https://support.nagios.com/forum/viewto ... 37&t=34467 ... But, No Luck...

attached are the outputs of some of the commands...

Any quick help would be appreciated... This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications...

Thanks!

Srinivas Mandalika

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Mon Dec 03, 2018 9:34 am
by mcapra
Does this machine have enough available disk space?

Code: Select all

df -h
Can you share the nxlog configuration from the remote machine that contains "InstaShare_Service_Server_Log"?

Also, can you share your Logstash configurations which can be obtained like so:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Mon Dec 03, 2018 10:43 am
by srinivasmandalika
Please find the requested files as attached...

Let me know if you need any further information...

Srinivas Mandalika

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Tue Dec 04, 2018 11:17 am
by srinivasmandalika
Any update on this please?

Thanks!

Srinivas Mandalika

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Tue Dec 04, 2018 11:37 am
by mcapra
srinivasmandalika wrote:This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications.
+1 for licensed support :) And if you do have a licensed support contract, I'd suggest putting urgent issues in the customer section or submitting them as an email.

There's no record within the Logstash logs of Logstash actually dying. The only errors I see are general insertion problems like this:

Code: Select all

{:timestamp=>"2018-12-03T03:20:05.594000-0500", :message=>"too many attempts at sending event. dropping: 2018-12-03T08:10:45.528Z 10.0.51.47 {\"EventReceivedTime\":\"2018-12-03 03:10:45\",\"SourceModuleName\":\"InstaShare_Service_Server_Log\",\"SourceModuleType\":\"im_file\",\"message\":\"[12/3/18 3:10:45:123 EST] 001897e9 com.ibm.wsspi.timedoperations.TimedOperationService          I Operation websphere.datasource.execute:jdbc/simonInstashareSvcDataSource:select applicatio0_.APPLICATION_SETTINGS_SKEY as APPLICAT1_0_, applicatio0_.APP_SET_NAME as APP_SET_2_0_, applicatio0_.APP_SET_VALUE as APP_SET_3_0_ from APPLICATION_SETTINGS applicatio0_ where applicatio0_.APP_SET_NAME like ? took 1.177 ms to complete\"}\r", :level=>:error}
It could be that Logstash is overloaded. It could be that ElasticSearch is overloaded. It's hard to tell with only a small snippet (~200 lines) of either logfile. Can you provide the Logstash and ElasticSearch logfiles in their entirety?

A lot of the failures seem to be coming from the InstaShare_Service_Server_Log record in your nxlog configuration. I'd start by disabling that and see if Logstash stability improves.

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Tue Dec 04, 2018 1:34 pm
by cdienger
Upping some settings for logstash can help as well. See https://support.nagios.com/kb/article/n ... g-576.html.

The logs requested will be included with profiles in recent versions(2.0.3 and up). Feel free PM me the profile which can be generated under Admin > System > System Status > Download System Profile.

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Tue Dec 04, 2018 1:50 pm
by srinivasmandalika
I tried to restart Logstash again and I see Logs are now being received normally...

Our Log server is up and running now...

I will keep an eye on this and will let you know if I face this problem again...

Also, we are in version 1.4.2... Do you have any documentation that can help me to upgrade to latest version? Any step by step procedure would be appreciated...

Thanks!

Re: Nagios Log Server is not collecting Logs from the machin

Posted: Tue Dec 04, 2018 5:24 pm
by cdienger
Sounds good. Here is a guide to help upgrade:

https://assets.nagios.com/downloads/nag ... Server.pdf