Nagios Stops reporting Every Night!

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
squirrely73
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Nagios Stops reporting Every Night!

Post by squirrely73 »

Stuck on trying to figure out why Nagios stops every night.
Nagios 3.5.1 Running on Red Hat 4.4.7

Every morning for awhile I come in and Nagios appears to be running and showing information but nothing is actually updating anymore.
When I issue the restart command I Get:

Running Config Check....done.
Stopping nagios: /etc/init.d/nagios: line 74: kill: (27789) - No such process
done.

Then Nagios starts back up and works normally and shows any current alerts and works fine the rest of the day.

I've been unable to find what specifically is stopping Nagios from updating in the middle of the night until a restart and was looking for some guidance in trying to resolve the issue.
User avatar
tacolover101
Posts: 432
Joined: Mon Apr 10, 2017 11:55 am

Re: Nagios Stops reporting Every Night!

Post by tacolover101 »

are you hosting nagios in a public cloud? something is killing the process, or it's dying somehow.

take a look at your nagios.log, you should see timestamps up until when it's killed off.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Nagios Stops reporting Every Night!

Post by npolovenko »

Thanks, @tacolover101!
@squirrely73 , Please upload you nagios log from:

Code: Select all

/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
squirrely73
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Post by squirrely73 »

npolovenko wrote: @squirrely73 , Please upload you nagios log from:

Code: Select all

/usr/local/nagios/var/nagios.log
and system log from /var/log/messages file.
For the /usr/local/nagios/var/nagios.log the result is "no such file or directory exists"
could it be in another location or loging isn't setup?
I recently received the duty or checking and working on Nagios from previous employee that left the company and as mentioned in previous posts know enough to be very dangerous.




for the messages file i'll have to wait until tomorrow to check as the file doesn't go back far enough to see time when it stopped.
I'll check and verify messages file tomorrow morning as i'll be able to see what it showed overnight when it quit.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Nagios Stops reporting Every Night!

Post by dwhitfield »

You should be able to find it using find / -name nagios.log.

You might as well go ahead and upload your nagios.cfg as well.

You may also want to go ahead and turn on debugging in the nagios.cfg: https://assets.nagios.com/downloads/nag ... gmain.html
squirrely73
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Post by squirrely73 »

I've attached my config file along with the highlights of the messages and log file:

It seems at Midnight i'm getting: Nov 7 00:00:00 wc-nagios nagios: Caught SIGSEGV, shutting down..
and that's when my monitoring ceases, everything looks fine if you go into the web interface to look at problems but it's not actually updating anything until reboot
nagios.cfg
(44.25 KiB) Downloaded 501 times
NagiosLogs.txt
(3.37 KiB) Downloaded 509 times
squirrely73
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Post by squirrely73 »

So doing research on "Caught SIGSEGV shutting down"

It seems to be related to a host in scheduled downtime i didn't think we had any in downtime but after looking through hosts i do see that 1 of them is scheduled for downtime and has been for sometime.

The person before me just must have left it in downtime and had been restarting every morning.

I'll check on that host and remove from downtime and see if the issue goes away
kyang

Re: Nagios Stops reporting Every Night!

Post by kyang »

Sounds good, let us know!
squirrely73
Posts: 22
Joined: Mon Jul 24, 2017 8:15 am

Re: Nagios Stops reporting Every Night!

Post by squirrely73 »

That took care of it, removing host from downtime fixed issues.

Thanks, feel free to close the topic
kyang

Re: Nagios Stops reporting Every Night!

Post by kyang »

Sounds great! I'll be closing this topic.

If you have any more questions, feel free to create another thread!

Thank you for using the Nagios Support Forum!
Locked