Page 1 of 3
httpd stops every hour
Posted: Tue Oct 19, 2021 8:32 am
by btsmnagios
We recently upgraded XI to 5.8.4, running on RedHat 7.4 64bit. Since the upgrade we observe the dashboard becoming unresponsive every hour for about 5-6 minutes before the httpd service restarts and the dashboard returns for another 60 minutes before the cycle repeats.
At the time this occurs we do see a spike in the number of processes from approximately 500 to over 800 just before we lose the dashboard. There is also appears to be higher CPU utilisation for mySQL during this time before the apache maxes out all cores at 100% and then the dashboard returns.
The only other thing we see which may be related is that we can no longer access the monitoring engine event queue. When clicking on this we are presented with a blank screen so cannot see if there is a high number of checks scheduled to be executed and causing a bottleneck. The server has 56 CPU cores and 64GB RAM.
There were no errors in the upgrade log and we dont see errors in /var/log/httpd/error_log
Re: httpd stops every hour
Posted: Tue Oct 19, 2021 11:24 am
by gsmith
Hi
Please check for cron jobs in the following places, and eaxmine them to see if any coincide
with the httpd lockup/restart:
cat /etc/crontab
cat /etc/cron.d/*
Redhat, CentOS, Amazon Linux, Suse
cat /var/spool/cron/*
Ubuntu/Debian
$ cat /var/spool/cron/crontabs/*
Next check var/log/messages for entries near those key times.
Please let me know what you find.
Thanks
Re: httpd stops every hour
Posted: Fri Oct 22, 2021 8:41 am
by btsmnagios
Have checked the various cron files we have and there is nothing set to run every hour. We have a few which are set to run at certain times of the day or night, some which run every 10 or 15 minutes and some which run every minute.
Re: httpd stops every hour
Posted: Fri Oct 22, 2021 2:35 pm
by gsmith
Hi,
Did you see anything in /var/log/messages?
Please send us your system profile.
1. Login to the Nagios XI GUI using a web browser.
2. Click the "Admin" > "System Profile" Menu
3. Click the "Download Profile" button
4. Save the profile.zip file and share the file in a private message and then
reply to this post to bring it up in the queue.
Thank you!
Re: httpd stops every hour
Posted: Tue Oct 26, 2021 9:40 am
by btsmnagios
profile.zip sent as requested
Re: httpd stops every hour
Posted: Tue Oct 26, 2021 9:28 pm
by gsmith
Hey btsmnagios,
I received the profile and will be loading it up in the morning.
Thanks
Re: httpd stops every hour
Posted: Wed Oct 27, 2021 2:45 pm
by gsmith
Hi
Could you look in /var/log:
Code: Select all
[root@gs-rhel8-23-84 /]# cd /var/log
[root@gs-rhel8-23-84 log]# ls -lh mess*
-rw------- 1 root root 8.4M Oct 27 14:44 messages
-rw-------. 1 root root 4.3M Oct 13 10:12 messages-20211013
-rw------- 1 root root 15M Oct 17 03:39 messages-20211017
-rw------- 1 root root 11M Oct 25 11:07 messages-20211025
[root@gs-rhel8-23-84 log]#
Please get one or two of the messages-XXXXXXXX files, zip them up, and send them to me like you
did the system profile. The one I received as part of the System Profile spans only minutes, I need to
get some data that spans a few hours.
Thank you
Re: httpd stops every hour
Posted: Mon Nov 01, 2021 4:08 am
by btsmnagios
Thank you. We're just sanitising our log files and will forward over.
Re: httpd stops every hour
Posted: Mon Nov 01, 2021 11:02 am
by gsmith
Sounds good - let me know when they're uploaded please.
Thanks
Re: httpd stops every hour
Posted: Fri Nov 19, 2021 5:23 am
by btsmnagios
Hi,
Our log has been uploaded. We need to send you the password for the file via separate means. Is there an email address I can forward this onto?