Page 1 of 1

Nagios Stops reporting Every Night!

Posted: Fri Nov 03, 2017 7:17 am
by squirrely73
Stuck on trying to figure out why Nagios stops every night.
Nagios 3.5.1 Running on Red Hat 4.4.7

Every morning for awhile I come in and Nagios appears to be running and showing information but nothing is actually updating anymore.
When I issue the restart command I Get:

Running Config Check....done.
Stopping nagios: /etc/init.d/nagios: line 74: kill: (27789) - No such process
done.

Then Nagios starts back up and works normally and shows any current alerts and works fine the rest of the day.

I've been unable to find what specifically is stopping Nagios from updating in the middle of the night until a restart and was looking for some guidance in trying to resolve the issue.

Re: Nagios Stops reporting Every Night!

Posted: Fri Nov 03, 2017 11:28 am
by tacolover101
are you hosting nagios in a public cloud? something is killing the process, or it's dying somehow.

take a look at your nagios.log, you should see timestamps up until when it's killed off.

Re: Nagios Stops reporting Every Night!

Posted: Fri Nov 03, 2017 1:13 pm
by npolovenko
Thanks, @tacolover101!
@squirrely73 , Please upload you nagios log from:

Code: Select all

/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.

Re: Nagios Stops reporting Every Night!

Posted: Mon Nov 06, 2017 10:17 am
by squirrely73
npolovenko wrote: @squirrely73 , Please upload you nagios log from:

Code: Select all

/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.
For the /usr/local/nagios/var/nagios.log the result is "no such file or directory exists"
could it be in another location or loging isn't setup?
I recently received the duty or checking and working on Nagios from previous employee that left the company and as mentioned in previous posts know enough to be very dangerous.




for the messages file i'll have to wait until tomorrow to check as the file doesn't go back far enough to see time when it stopped.
I'll check and verify messages file tomorrow morning as i'll be able to see what it showed overnight when it quit.

Re: Nagios Stops reporting Every Night!

Posted: Mon Nov 06, 2017 4:38 pm
by dwhitfield
You should be able to find it using find / -name nagios.log.

You might as well go ahead and upload your nagios.cfg as well.

You may also want to go ahead and turn on debugging in the nagios.cfg: https://assets.nagios.com/downloads/nag ... gmain.html

Re: Nagios Stops reporting Every Night!

Posted: Tue Nov 07, 2017 8:47 am
by squirrely73
I've attached my config file along with the highlights of the messages and log file:

It seems at Midnight i'm getting: Nov 7 00:00:00 wc-nagios nagios: Caught SIGSEGV, shutting down..
and that's when my monitoring ceases, everything looks fine if you go into the web interface to look at problems but it's not actually updating anything until reboot
nagios.cfg
(44.25 KiB) Downloaded 501 times
NagiosLogs.txt
(3.37 KiB) Downloaded 508 times

Re: Nagios Stops reporting Every Night!

Posted: Tue Nov 07, 2017 2:11 pm
by squirrely73
So doing research on "Caught SIGSEGV shutting down"

It seems to be related to a host in scheduled downtime i didn't think we had any in downtime but after looking through hosts i do see that 1 of them is scheduled for downtime and has been for sometime.

The person before me just must have left it in downtime and had been restarting every morning.

I'll check on that host and remove from downtime and see if the issue goes away

Re: Nagios Stops reporting Every Night!

Posted: Tue Nov 07, 2017 3:25 pm
by kyang
Sounds good, let us know!

Re: Nagios Stops reporting Every Night!

Posted: Fri Nov 10, 2017 1:26 pm
by squirrely73
That took care of it, removing host from downtime fixed issues.

Thanks, feel free to close the topic

Re: Nagios Stops reporting Every Night!

Posted: Fri Nov 10, 2017 1:38 pm
by kyang
Sounds great! I'll be closing this topic.

If you have any more questions, feel free to create another thread!

Thank you for using the Nagios Support Forum!