check_all_procs start reporting wrong every morning around 4

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
davidrk
Posts: 7
Joined: Sat Feb 01, 2020 9:44 pm
Location: Oak Ridge, TN

check_all_procs start reporting wrong every morning around 4

Post by davidrk »

Having a weird problem.

Using Nagios to monitor 2 servers, using check_all_procs, and every morning on both server, Nagios stops reporting the processes correctly.
Typically they are between 110-140, but every morning around 4am ET they shoot down to 2-3.

Reboot the servers, and all is well again until around 4am.

Nothing is running on the servers at that time to be causing an issue.

Anyone have any suggestions where to start looking?

Thanks,
David
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: check_all_procs start reporting wrong every morning arou

Post by Box293 »

What is in the logs?
/var/log/messages
/usr/local/nagios/var/nagios.log

You may need to enable debug logging on Nagios, try setting the debug level on and then restart Nagios.

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=-1/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart
Check the file /usr/local/nagios/var/nagios.debug

When you are finished this turns debugging off:

Code: Select all

sed -i 's/.*debug_level=.*/debug_level=0/g' /usr/local/nagios/etc/nagios.cfg
service nagios restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
davidrk
Posts: 7
Joined: Sat Feb 01, 2020 9:44 pm
Location: Oak Ridge, TN

Re: check_all_procs start reporting wrong every morning arou

Post by davidrk »

Only thing I am seeing close to that time is the log rotation on the server.

But don't see how that would be affection Nagios.
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: check_all_procs start reporting wrong every morning arou

Post by Box293 »

Did you enable debug logging ?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
davidrk
Posts: 7
Joined: Sat Feb 01, 2020 9:44 pm
Location: Oak Ridge, TN

Re: check_all_procs start reporting wrong every morning arou

Post by davidrk »

I did enable it, and it failed on 2 of the 3 servers being monitored.
All are running CentOS 7 and CWP.

Can I PM you the debug log to look at?
benjaminsmith
Posts: 5324
Joined: Wed Aug 22, 2018 4:39 pm
Location: saint paul

Re: check_all_procs start reporting wrong every morning arou

Post by benjaminsmith »

Hello David,
Can I PM you the debug log to look at?
Certainly. Please send it to me in PM and I can share this with Box293. Thanks.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.

Be sure to check out our Knowledgebase for helpful articles and solutions!
davidrk
Posts: 7
Joined: Sat Feb 01, 2020 9:44 pm
Location: Oak Ridge, TN

Re: check_all_procs start reporting wrong every morning arou

Post by davidrk »

I don't think this is a Nagios XI problem, since the 3rd server which happens to be a Non Pro CWP server is working correctly with stats.

Attached is the debug file, if someone can look just to double check it.
ATL3 and ATL6 have the problem, and have to be rebooted to start reporting correctly again, but EWR3 is OK.


Thanks,
David
Attachments
nagios.debug.txt
(600.34 KiB) Downloaded 180 times
davidrk
Posts: 7
Joined: Sat Feb 01, 2020 9:44 pm
Location: Oak Ridge, TN

Re: check_all_procs start reporting wrong every morning arou

Post by davidrk »

Here is the graph from Nagios.

Weird.
All is well after a manually reboot of the server.
Attachments
Nagios Problem.jpg
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: check_all_procs start reporting wrong every morning arou

Post by Box293 »

This completely seems like an issue on your servers and nothing to do with Nagios.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked