httpd stops every hour

[btsmnagios]

We recently upgraded XI to 5.8.4, running on RedHat 7.4 64bit. Since the upgrade we observe the dashboard becoming unresponsive every hour for about 5-6 minutes before the httpd service restarts and the dashboard returns for another 60 minutes before the cycle repeats.
At the time this occurs we do see a spike in the number of processes from approximately 500 to over 800 just before we lose the dashboard. There is also appears to be higher CPU utilisation for mySQL during this time before the apache maxes out all cores at 100% and then the dashboard returns.

The only other thing we see which may be related is that we can no longer access the monitoring engine event queue. When clicking on this we are presented with a blank screen so cannot see if there is a high number of checks scheduled to be executed and causing a bottleneck. The server has 56 CPU cores and 64GB RAM.

There were no errors in the upgrade log and we dont see errors in /var/log/httpd/error_log

[gsmith]

Hi

Please check for cron jobs in the following places, and eaxmine them to see if any coincide
with the httpd lockup/restart:

cat /etc/crontab

cat /etc/cron.d/*

Redhat, CentOS, Amazon Linux, Suse
cat /var/spool/cron/*

Ubuntu/Debian
$ cat /var/spool/cron/crontabs/*

Next check var/log/messages for entries near those key times.

Please let me know what you find.

Thanks

[btsmnagios]

Have checked the various cron files we have and there is nothing set to run every hour. We have a few which are set to run at certain times of the day or night, some which run every 10 or 15 minutes and some which run every minute.

[gsmith]

Hi,

Did you see anything in /var/log/messages?

Please send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share the file in a private message and then
reply to this post to bring it up in the queue.

Thank you!

[btsmnagios]

profile.zip sent as requested

[gsmith]

Hey btsmnagios,

I received the profile and will be loading it up in the morning.

Thanks

[gsmith]

Hi

Could you look in /var/log:

Code: Select all

[root@gs-rhel8-23-84 /]# cd /var/log
[root@gs-rhel8-23-84 log]# ls -lh mess*
-rw-------  1 root root 8.4M Oct 27 14:44 messages
-rw-------. 1 root root 4.3M Oct 13 10:12 messages-20211013
-rw-------  1 root root  15M Oct 17 03:39 messages-20211017
-rw-------  1 root root  11M Oct 25 11:07 messages-20211025
[root@gs-rhel8-23-84 log]#

Please get one or two of the messages-XXXXXXXX files, zip them up, and send them to me like you
did the system profile. The one I received as part of the System Profile spans only minutes, I need to
get some data that spans a few hours.

Thank you

[btsmnagios]

Thank you. We're just sanitising our log files and will forward over.

[gsmith]

Sounds good - let me know when they're uploaded please.

Thanks

[btsmnagios]

Hi,

Our log has been uploaded. We need to send you the password for the file via separate means. Is there an email address I can forward this onto?