Database Maintenance showing RED very often

[mejokj]

Hi Team,

We have observed that very frequently the database maintenance under admin tab showing RED. We have done below steps to clear it. Please let us know if we avoid these issue permanently.

Attached the error and profile file.

steps done to resolve this issue
++++++++++++++++++++++++

service nagios stop
service ndo2db stop
service mysqld stop
service crond stop
service httpd stop
killall -9 nagios
killall -9 ndo2db
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
service ndo2db start
service nagios start
service mysqld start
service crond start
service httpd start
+++++++++++++++++++++++++++

Some logs for reference
++++++++++++++++++

[root@nagios html]# cat /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 7 * * * root /root/scripts/autopostgresqlbackup > /dev/null 2>&1

* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/recurring_downtime.php >> /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1

[root@nagios html]#

[root@nagios html]# ls -al /usr/local/nagiosxi/cron/
total 200
drwxr-xr-x 2 root nagios 4096 Sep 25 16:34 .
drwxr-xr-x 10 root nagios 4096 Apr 30 2019 ..
-rwxr-xr-x 1 nagios nagios 1456 Sep 25 16:34 cleaner.php
-rwxr-xr-x 1 nagios nagios 45773 Sep 25 16:34 cmdsubsys.php
-rwxr-xr-x 1 nagios nagios 16169 Sep 25 16:34 dbmaint.php
-rw-r--r-- 1 nagios nagios 33047 Sep 25 16:34 deadpool.php
-rwxr-xr-x 1 nagios nagios 4143 Sep 25 16:34 event_handler.php
-rwxr-xr-x 1 nagios nagios 4884 Sep 25 16:34 eventman.php
-rwxr-xr-x 1 nagios nagios 2336 Sep 25 16:34 feedproc.php
-rwxr-xr-x 1 nagios nagios 2166 Sep 25 16:34 nom.php
-rwxr-xr-x 1 nagios nagios 21819 Sep 25 16:34 perfdataproc.php
-rwxr-xr-x 1 nagios nagios 19624 Sep 25 16:34 recurring_downtime.php
-rwxr-xr-x 1 nagios nagios 1587 Sep 25 16:34 reportengine.php
-rw-r--r-- 1 nagios nagios 1966 Sep 25 16:34 snmptt_service_results.php
-rwxr-xr-x 1 nagios nagios 11712 Sep 25 16:34 sysstat.php
You have mail in /var/spool/mail/root
[root@nagios html]# md5sum /usr/local/nagiosxi/cron/dbmaint.php
a465c2e51318186dae9481ee2b70842b /usr/local/nagiosxi/cron/dbmaint.php
[root@nagios html]#


[root@nagios ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 490G 68G 402G 15% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 772M 7.0G 10% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 976M 197M 713M 22% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/1000
tmpfs 1.6G 0 1.6G 0% /run/user/0
tmpfs 1.6G 0 1.6G 0% /run/user/1001

[root@nagios ~]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/centos-root 32571392 6336460 26234932 20% /
devtmpfs 2029237 422 2028815 1% /dev
tmpfs 2032194 1 2032193 1% /dev/shm
tmpfs 2032194 1806 2030388 1% /run
tmpfs 2032194 16 2032178 1% /sys/fs/cgroup
/dev/sda1 65536 351 65185 1% /boot
tmpfs 2032194 1 2032193 1% /run/user/1000
tmpfs 2032194 1 2032193 1% /run/user/0
tmpfs 2032194 1 2032193 1% /run/user/1001
[root@nagios ~]#

[tgriep]

I found a too many connections error to the MYSQL database which would cause the issue you are seeing for the DBMaint process.
Follow this KB article to fix that.
https://support.nagios.com/kb/article/n ... s-513.html

Run this to stop the processes and repait the MYSQL database.

Code: Select all

systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop crond
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
systemctl restart mariadb
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill python
systemctl restart httpd
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

I found some other issues in the Profile.
There is a missing SNMP MIB file.
Attached to the post is a SNMP-TARGET-MIB.txt file.
Upload it to the Nagios server using the Admin > Manage MIBs menu.
That should stop those messages.

Let us know if the DBMaint process it running better.