Nagios Stops reporting Every Night!

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

Nagios Stops reporting Every Night!

Postby squirrely73 » Fri Nov 03, 2017 7:17 am

Stuck on trying to figure out why Nagios stops every night.
Nagios 3.5.1 Running on Red Hat 4.4.7

Every morning for awhile I come in and Nagios appears to be running and showing information but nothing is actually updating anymore.
When I issue the restart command I Get:

Running Config Check....done.
Stopping nagios: /etc/init.d/nagios: line 74: kill: (27789) - No such process
done.

Then Nagios starts back up and works normally and shows any current alerts and works fine the rest of the day.

I've been unable to find what specifically is stopping Nagios from updating in the middle of the night until a restart and was looking for some guidance in trying to resolve the issue.
squirrely73
 
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Postby tacolover101 » Fri Nov 03, 2017 11:28 am

are you hosting nagios in a public cloud? something is killing the process, or it's dying somehow.

take a look at your nagios.log, you should see timestamps up until when it's killed off.
User avatar
tacolover101
 
Posts: 326
Joined: Mon Apr 10, 2017 11:55 am

Re: Nagios Stops reporting Every Night!

Postby npolovenko » Fri Nov 03, 2017 1:13 pm

Thanks, @tacolover101!
@squirrely73 , Please upload you nagios log from:
Code: Select all
/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.
User avatar
npolovenko
 
Posts: 367
Joined: Mon May 15, 2017 5:00 pm

Re: Nagios Stops reporting Every Night!

Postby squirrely73 » Mon Nov 06, 2017 10:17 am

npolovenko wrote:@squirrely73 , Please upload you nagios log from:
Code: Select all
/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.


For the /usr/local/nagios/var/nagios.log the result is "no such file or directory exists"
could it be in another location or loging isn't setup?
I recently received the duty or checking and working on Nagios from previous employee that left the company and as mentioned in previous posts know enough to be very dangerous.




for the messages file i'll have to wait until tomorrow to check as the file doesn't go back far enough to see time when it stopped.
I'll check and verify messages file tomorrow morning as i'll be able to see what it showed overnight when it quit.
squirrely73
 
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Postby dwhitfield » Mon Nov 06, 2017 4:38 pm

You should be able to find it using find / -name nagios.log.

You might as well go ahead and upload your nagios.cfg as well.

You may also want to go ahead and turn on debugging in the nagios.cfg: https://assets.nagios.com/downloads/nag ... gmain.html
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
dwhitfield
The Doctor
 
Posts: 3956
Joined: Wed Sep 21, 2016 10:29 am
Location: Nagios Enterprises, LLC

Re: Nagios Stops reporting Every Night!

Postby squirrely73 » Tue Nov 07, 2017 8:47 am

I've attached my config file along with the highlights of the messages and log file:

It seems at Midnight i'm getting: Nov 7 00:00:00 wc-nagios nagios: Caught SIGSEGV, shutting down..
and that's when my monitoring ceases, everything looks fine if you go into the web interface to look at problems but it's not actually updating anything until reboot

nagios.cfg
(44.25 KiB) Downloaded 9 times
NagiosLogs.txt
(3.37 KiB) Downloaded 9 times
squirrely73
 
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Postby squirrely73 » Tue Nov 07, 2017 2:11 pm

So doing research on "Caught SIGSEGV shutting down"

It seems to be related to a host in scheduled downtime i didn't think we had any in downtime but after looking through hosts i do see that 1 of them is scheduled for downtime and has been for sometime.

The person before me just must have left it in downtime and had been restarting every morning.

I'll check on that host and remove from downtime and see if the issue goes away
squirrely73
 
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Postby kyang » Tue Nov 07, 2017 3:25 pm

Sounds good, let us know!
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
kyang
 
Posts: 673
Joined: Tue Jul 25, 2017 3:35 pm

Re: Nagios Stops reporting Every Night!

Postby squirrely73 » Fri Nov 10, 2017 1:26 pm

That took care of it, removing host from downtime fixed issues.

Thanks, feel free to close the topic
squirrely73
 
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Postby kyang » Fri Nov 10, 2017 1:38 pm

Sounds great! I'll be closing this topic.

If you have any more questions, feel free to create another thread!

Thank you for using the Nagios Support Forum!
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
kyang
 
Posts: 673
Joined: Tue Jul 25, 2017 3:35 pm


Return to Nagios Core

Who is online

Users browsing this forum: Bing [Bot] and 25 guests