Nagios Log Server is not collecting Logs from the machines

This board serves as an open discussion and support collaboration point for Nagios Log Server. NOTE: Nagios Log Server customers should use the Customer Support forum to obtain expedited support.

Nagios Log Server is not collecting Logs from the machines

Postby srinivasmandalika » Mon Dec 03, 2018 3:40 am

Hello,

We are facing issue with our Nagios Log Server of not collecting logs from the servers...

First I see that Logstash process is dying frequently even after restart and followed procedure mentioned in https://support.nagios.com/kb/article/n ... g-576.html

Then, I see the home page states it receiving logs from only 1 host... I followed https://support.nagios.com/kb/article.php?id=38 ... But, No Luck

I followed viewtopic.php?f=37&t=34467 ... But, No Luck...

attached are the outputs of some of the commands...

Any quick help would be appreciated... This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications...

Thanks!

Srinivas Mandalika
Attachments
nagios error.txt
(29.82 KiB) Downloaded 11 times
elasticsearch logs.txt
(33.9 KiB) Downloaded 9 times
srinivasmandalika
 
Posts: 37
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby mcapra » Mon Dec 03, 2018 9:34 am

Does this machine have enough available disk space?
Code: Select all
df -h


Can you share the nxlog configuration from the remote machine that contains "InstaShare_Service_Server_Log"?

Also, can you share your Logstash configurations which can be obtained like so:
Code: Select all
grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*
Former Nagios employee
http://www.mcapra.com/
User avatar
mcapra
 
Posts: 3462
Joined: Thu May 05, 2016 3:54 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby srinivasmandalika » Mon Dec 03, 2018 10:43 am

Please find the requested files as attached...

Let me know if you need any further information...

Srinivas Mandalika
Attachments
nxlog_waslibprd1.conf
(2.83 KiB) Downloaded 11 times
Logstash Configuration and disk space.txt
(13.29 KiB) Downloaded 10 times
srinivasmandalika
 
Posts: 37
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby srinivasmandalika » Tue Dec 04, 2018 11:17 am

Any update on this please?

Thanks!

Srinivas Mandalika
srinivasmandalika
 
Posts: 37
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby mcapra » Tue Dec 04, 2018 11:37 am

srinivasmandalika wrote:This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications.


+1 for licensed support :) And if you do have a licensed support contract, I'd suggest putting urgent issues in the customer section or submitting them as an email.

There's no record within the Logstash logs of Logstash actually dying. The only errors I see are general insertion problems like this:
Code: Select all
{:timestamp=>"2018-12-03T03:20:05.594000-0500", :message=>"too many attempts at sending event. dropping: 2018-12-03T08:10:45.528Z 10.0.51.47 {\"EventReceivedTime\":\"2018-12-03 03:10:45\",\"SourceModuleName\":\"InstaShare_Service_Server_Log\",\"SourceModuleType\":\"im_file\",\"message\":\"[12/3/18 3:10:45:123 EST] 001897e9 com.ibm.wsspi.timedoperations.TimedOperationService          I Operation websphere.datasource.execute:jdbc/simonInstashareSvcDataSource:select applicatio0_.APPLICATION_SETTINGS_SKEY as APPLICAT1_0_, applicatio0_.APP_SET_NAME as APP_SET_2_0_, applicatio0_.APP_SET_VALUE as APP_SET_3_0_ from APPLICATION_SETTINGS applicatio0_ where applicatio0_.APP_SET_NAME like ? took 1.177 ms to complete\"}\r", :level=>:error}


It could be that Logstash is overloaded. It could be that ElasticSearch is overloaded. It's hard to tell with only a small snippet (~200 lines) of either logfile. Can you provide the Logstash and ElasticSearch logfiles in their entirety?

A lot of the failures seem to be coming from the InstaShare_Service_Server_Log record in your nxlog configuration. I'd start by disabling that and see if Logstash stability improves.
Former Nagios employee
http://www.mcapra.com/
User avatar
mcapra
 
Posts: 3462
Joined: Thu May 05, 2016 3:54 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby cdienger » Tue Dec 04, 2018 1:34 pm

Upping some settings for logstash can help as well. See https://support.nagios.com/kb/article/n ... g-576.html.

The logs requested will be included with profiles in recent versions(2.0.3 and up). Feel free PM me the profile which can be generated under Admin > System > System Status > Download System Profile.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 2191
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server is not collecting Logs from the machin

Postby srinivasmandalika » Tue Dec 04, 2018 1:50 pm

I tried to restart Logstash again and I see Logs are now being received normally...

Our Log server is up and running now...

I will keep an eye on this and will let you know if I face this problem again...

Also, we are in version 1.4.2... Do you have any documentation that can help me to upgrade to latest version? Any step by step procedure would be appreciated...

Thanks!
srinivasmandalika
 
Posts: 37
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Postby cdienger » Tue Dec 04, 2018 5:24 pm

Sounds good. Here is a guide to help upgrade:

https://assets.nagios.com/downloads/nag ... Server.pdf
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
cdienger
Support Tech
 
Posts: 2191
Joined: Tue Feb 07, 2017 11:26 am


Return to Nagios Log Server

Who is online

Users browsing this forum: No registered users and 3 guests