host check pending, performance data redux

[pnewlon]

Hosts are checking, performance data is processing, I have more hosts that are showing 'check pending' than not and only a smattering of performance data. The first time this happened, no hosts/services were being checked and all perf graphs missing. This time all (that I can tell) host and service checks are being done but the web interface says pending and 'You are not authorized to access this feature' on some (most) of the performance graph panes.

I have a 100MB ramdisk that filled up yesterday. I changed it to 750MB this morning and rebooted the server. At the moment it is only 3% full after three hours of the server running.

tmpfs 750M 19M 732M 3% /var/nagiosramdisk

[root@LPNAGV04 00498_OB02-MP]# pwd
/usr/local/nagios/share/perfdata/00498_OB02-MP

[root@LPNAGV04 00498_OB02-MP]# ls -lt
total 2272
-rwxrwxr-x 1 nagios nagios 768224 Apr 7 10:45 Ping.rrd
-rw-rw-rw- 1 nagios nagios 2787 Apr 7 10:45 Ping.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 7 10:43 logicalDiskFreeSpace.rrd
-rw-rw-rw- 1 nagios nagios 2595 Apr 7 10:43 logicalDiskFreeSpace.xml
-rwxrwxr-x 1 nagios nagios 384952 Apr 7 10:41 memAvailMB.rrd
-rw-rw-rw- 1 nagios nagios 2366 Apr 7 10:41 memAvailMB.xml
-rwxrwxr-x 1 nagios nagios 768224 Apr 7 10:39 _HOST_.rrd
-rw-rw-rw- 1 nagios nagios 2676 Apr 7 10:39 _HOST_.xml

[abrist]

Load is spiking. Lets check a few things:

Code: Select all

tail -20 /var/log/mysqld.log
tail -20 /usr/local/nagios/var/npcd.log
ps -aef
df -i
df -h

[pnewlon]

I guess I have to 'repair the mysql database' again..... did that on Friday. Ugh.


140407 11:36:49 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_statehistory' is marked as crashed and last (automatic?) repair failed

[04-07-2014 11:37:00] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:37:15] NPCD: Processing file '1396885015.perfdata.host'
[04-07-2014 11:37:15] NPCD: Processing file '1396885015.perfdata.service'
[04-07-2014 11:37:15] NPCD: Processing file '1396885030.perfdata.service'
[04-07-2014 11:37:15] NPCD: Processing file '1396885030.perfdata.host'
[04-07-2014 11:37:15] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:37:30] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:37:45] NPCD: Processing file '1396885060.perfdata.service'
[04-07-2014 11:37:45] NPCD: Processing file '1396885045.perfdata.service'
[04-07-2014 11:37:45] NPCD: Processing file '1396885045.perfdata.host'
[04-07-2014 11:37:45] NPCD: Processing file '1396885060.perfdata.host'
[04-07-2014 11:37:47] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:38:02] NPCD: Processing file '1396885075.perfdata.host'
[04-07-2014 11:38:02] NPCD: Processing file '1396885075.perfdata.service'
[04-07-2014 11:38:02] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:38:17] NPCD: Processing file '1396885090.perfdata.host'
[04-07-2014 11:38:17] NPCD: Processing file '1396885090.perfdata.service'
[04-07-2014 11:38:18] NPCD: No more files to process... waiting for 15 seconds
[04-07-2014 11:38:33] NPCD: Processing file '1396885105.perfdata.host'
[04-07-2014 11:38:33] NPCD: Processing file '1396885105.perfdata.service'

[root@LPNAGV04 00498_OB02-MP]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
494832 90658 404174 19% /
tmpfs 1007596 1 1007595 1% /dev/shm
/dev/sda1 128016 44 127972 1% /boot
/dev/sdb1 5242880 26829 5216051 1% /usr/local
/dev/sdc1 5242880 685 5242195 1% /store
tmpfs 1007596 473 1007123 1% /var/nagiosramdisk


[root@LPNAGV04 00498_OB02-MP]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
7.5G 3.0G 4.1G 42% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 50M 410M 11% /boot
/dev/sdb1 79G 7.7G 68G 11% /usr/local
/dev/sdc1 79G 24G 52G 32% /store
tmpfs 750M 20M 731M 3% /var/nagiosramdisk

[abrist]

When the ramdisk filled up, you most likely had issues writing to db, creating lock files, etc. Repair the db and then restart the necessary services:

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios
/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql
service mysqld restart
service npcd restart
service ndo2db restart
service nagios restart

[pnewlon]

I truncated notifications, statehistory, logentries then repaired nagios and nagiosql. I have lots of green again.

[lmiltchev]

Sounds good! Can we lock the topic and mark it as "resolved"?

[pnewlon]

This one can be closed. opening another