Nagios Log Server is not collecting Logs from the machines

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
srinivasmandalika
Posts: 43
Joined: Thu Oct 20, 2016 4:09 pm

Nagios Log Server is not collecting Logs from the machines

Post by srinivasmandalika »

Hello,

We are facing issue with our Nagios Log Server of not collecting logs from the servers...

First I see that Logstash process is dying frequently even after restart and followed procedure mentioned in https://support.nagios.com/kb/article/n ... g-576.html

Then, I see the home page states it receiving logs from only 1 host... I followed https://support.nagios.com/kb/article.php?id=38 ... But, No Luck

I followed https://support.nagios.com/forum/viewto ... 37&t=34467 ... But, No Luck...

attached are the outputs of some of the commands...

Any quick help would be appreciated... This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications...

Thanks!

Srinivas Mandalika
You do not have the required permissions to view the files attached to this post.
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Nagios Log Server is not collecting Logs from the machin

Post by mcapra »

Does this machine have enough available disk space?

Code: Select all

df -h
Can you share the nxlog configuration from the remote machine that contains "InstaShare_Service_Server_Log"?

Also, can you share your Logstash configurations which can be obtained like so:

Code: Select all

grep '' /usr/local/nagioslogserver/logstash/etc/conf.d/*
Former Nagios employee
https://www.mcapra.com/
srinivasmandalika
Posts: 43
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Post by srinivasmandalika »

Please find the requested files as attached...

Let me know if you need any further information...

Srinivas Mandalika
You do not have the required permissions to view the files attached to this post.
srinivasmandalika
Posts: 43
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Post by srinivasmandalika »

Any update on this please?

Thanks!

Srinivas Mandalika
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Nagios Log Server is not collecting Logs from the machin

Post by mcapra »

srinivasmandalika wrote:This is a Production environment server and Application teams will rely on this instance to report them any errors in their applications.
+1 for licensed support :) And if you do have a licensed support contract, I'd suggest putting urgent issues in the customer section or submitting them as an email.

There's no record within the Logstash logs of Logstash actually dying. The only errors I see are general insertion problems like this:

Code: Select all

{:timestamp=>"2018-12-03T03:20:05.594000-0500", :message=>"too many attempts at sending event. dropping: 2018-12-03T08:10:45.528Z 10.0.51.47 {\"EventReceivedTime\":\"2018-12-03 03:10:45\",\"SourceModuleName\":\"InstaShare_Service_Server_Log\",\"SourceModuleType\":\"im_file\",\"message\":\"[12/3/18 3:10:45:123 EST] 001897e9 com.ibm.wsspi.timedoperations.TimedOperationService          I Operation websphere.datasource.execute:jdbc/simonInstashareSvcDataSource:select applicatio0_.APPLICATION_SETTINGS_SKEY as APPLICAT1_0_, applicatio0_.APP_SET_NAME as APP_SET_2_0_, applicatio0_.APP_SET_VALUE as APP_SET_3_0_ from APPLICATION_SETTINGS applicatio0_ where applicatio0_.APP_SET_NAME like ? took 1.177 ms to complete\"}\r", :level=>:error}
It could be that Logstash is overloaded. It could be that ElasticSearch is overloaded. It's hard to tell with only a small snippet (~200 lines) of either logfile. Can you provide the Logstash and ElasticSearch logfiles in their entirety?

A lot of the failures seem to be coming from the InstaShare_Service_Server_Log record in your nxlog configuration. I'd start by disabling that and see if Logstash stability improves.
Former Nagios employee
https://www.mcapra.com/
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server is not collecting Logs from the machin

Post by cdienger »

Upping some settings for logstash can help as well. See https://support.nagios.com/kb/article/n ... g-576.html.

The logs requested will be included with profiles in recent versions(2.0.3 and up). Feel free PM me the profile which can be generated under Admin > System > System Status > Download System Profile.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
srinivasmandalika
Posts: 43
Joined: Thu Oct 20, 2016 4:09 pm

Re: Nagios Log Server is not collecting Logs from the machin

Post by srinivasmandalika »

I tried to restart Logstash again and I see Logs are now being received normally...

Our Log server is up and running now...

I will keep an eye on this and will let you know if I face this problem again...

Also, we are in version 1.4.2... Do you have any documentation that can help me to upgrade to latest version? Any step by step procedure would be appreciated...

Thanks!
User avatar
cdienger
Support Tech
Posts: 5045
Joined: Tue Feb 07, 2017 11:26 am

Re: Nagios Log Server is not collecting Logs from the machin

Post by cdienger »

Sounds good. Here is a guide to help upgrade:

https://assets.nagios.com/downloads/nag ... Server.pdf
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked