Monitoring Engine Event Queue display no activity

[bosecorp]

My monitor engine event queue since today is acting up, or at least the system doesn't seem to be processing checks.

* I tried checking the DB Health
* I dont see any errors on on the LOGs, including the gearmand and nagios.log
* I tried restarting nagios and the database.
* File Systems look OK.
* tried running a repair database script, everything came back clean.
* rebooted the server.
* on the nagios.log file, I see a lot passive check being process but not a lot active checks
* all workers seem OK

any of the above don;t seem to help

gearman_top does seem to report some activity, but I don't see a lot activity.

----------------------------------------------------------------------------
check_results | 2 | 0 | 0
eventhandler | 29 | 0 | 0
host | 29 | 0 | 0
hostgroup_gearman_dca1 | 20 | 0 | 0
hostgroup_gearman_dce1 | 53 | 0 | 2
hostgroup_gearman_dcn1 | 43 | 0 | 1
hostgroup_gearman_dcn2 | 29 | 0 | 2
hostgroup_gearman_dcn3 | 25 | 0 | 0
hostgroup_gearman_hk1 | 41 | 0 | 2
hostgroup_gearman_mi1 | 20 | 0 | 0
hostgroup_gearman_my1 | 20 | 0 | 0
hostgroup_gearman_sl1 | 25 | 0 | 0
hostgroup_gearman_tj1 | 25 | 0 | 0
service | 29 | 0 | 0
servicegroup_gearman_mrtg | 29 | 0 | 0
worker_gearmandca1 | 1 | 0 | 0
worker_gearmandce1 | 1 | 0 | 0
worker_gearmandcn1 | 1 | 0 | 0
worker_gearmandcn2 | 1 | 0 | 0
worker_gearmandcn3 | 1 | 0 | 0
worker_gearmanhk1 | 1 | 0 | 0
worker_gearmanmi1 | 1 | 0 | 0
worker_gearmanmy1 | 1 | 0 | 0
----------------------------------------------------------------------------

[Box293]

Can you please run these commands and send us the output:

Code: Select all

ps -ef | grep nagios.cfg
ipcs -q
df -h
df -i

[bosecorp]

I was able to restore nagios. restarted gearmand. however I tried that before.

for sure I think the problem will came back, but for now I am good

you can close the ticket, but I might need to re-open it again

root@nagmonus1:(02-12 07:29): /root
# ps -ef | grep nagios.cfg
nagios 2077 1 10 01:02 ? 00:41:40 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 2240 2077 0 01:03 ? 00:00:02 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 27899 27834 0 07:29 pts/0 00:00:00 grep nagios.cfg
root@nagmonus1:(02-12 07:29): /root
# ipcs -q
df -h
df -i
------ Message Queues --------
key msqid owner perms used-bytes messages

root@nagmonus1:(02-12 07:29): /root
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lvroot 2.0G 1.1G 811M 58% /
tmpfs 15G 0 15G 0% /dev/shm
/dev/sda1 243M 49M 181M 22% /boot
/dev/mapper/rootvg-lvopt 2.0G 92M 1.8G 5% /opt
/dev/mapper/rootvg-lvtmp 6.9G 77M 6.5G 2% /tmp
/dev/mapper/rootvg-lvusers 4.0G 137M 3.7G 4% /users
/dev/mapper/rootvg-lvusr 7.9G 4.3G 3.3G 57% /usr
/dev/mapper/rootvg-lvvar 30G 9.9G 19G 36% /var
/dev/mapper/vgapp-lvapp 49G 4.1G 42G 9% /app
/dev/mapper/vgapp-lvstore 69G 30G 36G 46% /store
/dev/mapper/vgapp-lvlocalnagios 128G 79G 44G 65% /usr/local/nagios
/dev/mapper/vgapp-lvmysql 69G 2.6G 63G 4% /var/lib/mysql
/dev/mapper/vgapp-lvmodgearlog 20G 173M 19G 1% /var/log/mod_gearman
/dev/mapper/vgapp-lvgearlog 20G 174M 19G 1% /var/log/gearmand
tmpfs 2.0G 172M 1.9G 9% /var/nagiosramdisk
solid:/Home314/el872784 3.3T 2.9T 408G 88% /home/el872784
root@nagmonus1:(02-12 07:29): /root
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/rootvg-lvroot 131072 12009 119063 10% /
tmpfs 3867615 1 3867614 1% /dev/shm
/dev/sda1 64000 46 63954 1% /boot
/dev/mapper/rootvg-lvopt 131072 65 131007 1% /opt
/dev/mapper/rootvg-lvtmp 458752 987 457765 1% /tmp
/dev/mapper/rootvg-lvusers 262144 49 262095 1% /users
/dev/mapper/rootvg-lvusr 524288 160140 364148 31% /usr
/dev/mapper/rootvg-lvvar 1966080 36516 1929564 2% /var
/dev/mapper/vgapp-lvapp 3211264 43 3211221 1% /app
/dev/mapper/vgapp-lvstore 4587520 400 4587120 1% /store
/dev/mapper/vgapp-lvlocalnagios 8519680 82716 8436964 1% /usr/local/nagios
/dev/mapper/vgapp-lvmysql 4587520 533 4586987 1% /var/lib/mysql
/dev/mapper/vgapp-lvmodgearlog 1310720 21 1310699 1% /var/log/mod_gearman
/dev/mapper/vgapp-lvgearlog 1310720 12 1310708 1% /var/log/gearmand
tmpfs 3867615 63824 3803791 2% /var/nagiosramdisk
solid:/Home314/el872784 15805223 6753260 9051963 43% /home/el87278

[lmiltchev]

you can close the ticket, but I might need to re-open it again

Send any of us on the Nagios Support team a PM if you want to reopen the thread. Thanks!