httpd stops every hour

[gsmith]

Hi

Please use [email protected]

Thanks

[btsmnagios]

Password sent.

Many thanks for your patience.

[gsmith]

Hi

Thanks - I am able to open the messages file.

Will let you know when I find something.

- G

[gsmith]

Hi

We are suspecting there is something going on with the database. The DBMaint script runs hourly.

To test this please go to Admin, Performance Settings and go to the Databases tab. Scroll to the bottom,
In the "CCM Database" section change the "Optimize Interval" from 60 to 30

Now wait 90 to 120 minutes to see if httpd restarts every 30 minutes. If it does then we need to look
for corrupted/damaged db tables.

Let me know what the results are. You can change the "Optimize Interval" back to 60 after this test.

Thanks

[btsmnagios]

Hi,

There was a suspicion this may be db related as we had to perform db repair twice after out last XI upgrade.

No difference other than the first time after the optimization interval was changed to 30, we saw the number of processes spike to 950 instead of the usual 800ish. Interface came back up after about 5 minutes again then the same behaviour repeated again after about another hour with processes spiking to 800.
To check overnight I have increased the Optimize Interval from 60 to 90 minutes for the XI database and increased the same value to 75 minutes for the NDO database to see if we have any sort of change and will report back in the morning.

[gsmith]

Ok - let me know what happens.

Thanks you

[btsmnagios]

Hi,

Appears to be the XI Database causing the running processes spike. After increasing to 90 minutes we see the spikes happening after 90 minutes and the web interface becoming unreachable during the same times.

[gsmith]

Hi

Let's try to stop everything and do a database repair and restart. Please log in as root and run the following:

Code: Select all

systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop crond
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then systemctl stop postgresql; fi;
systemctl restart mariadb
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
rm -f /usr/local/nagiosxi/tmp/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill python
if grep --quiet pgsql /usr/local/nagiosxi/html/config.inc.php; then service postgresql start; fi;
systemctl restart httpd
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

I realize that you will probably have to do this tonight or over the weekend, let me know what happens.

Thanks

[btsmnagios]

Hi,

Have run the commands this afternoon and initially all seemed ok however we've now reverted back to the spike in processes and the UI drops out again.

[btsmnagios]

further to this it actually appears to be worse now, we're losing the UI every few minutes.