Nagios database seems hosed

[snapon_admin]

so I came in today unable to access our Nagios server (could ping it, but the web URL wouldn't load and it was inaccessible via SSH). I had one of our VM admins reboot it for me and it came back up but the page only showed the "your databases are broke, run the database repair script" message all over the screen so I did that. After the script ran the DBs still seemed jacked so I applied a config which didn't fix it. So I then rebooted the server from an SSH session and it booted back up and that message was gone. now I'm getting this message:

Code: Select all

SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus'

I tried restoring from the most recent backup (yesterdays) and got this error at the end of the backup restore:

Code: Select all

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Error restoring MySQL database 'nagios' - check the password in this script!

What do?

[tgriep]

Can you post the output of this command?

Code: Select all

tail -50 /var/log/mysqld.log

Here is another method of repairing the mysql database. You can try it to see if it repairs the database.

Code: Select all

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php

[snapon_admin]

Code: Select all

[root@keno-ngos-01-pv nagios]# tail -50 /var/log/mysqld.log
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:22 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:18 [Note] /usr/libexec/mysqld: Normal shutdown

150831 12:08:18 [Note] Event Scheduler: Purging the queue. 0 events
150831 12:08:20  InnoDB: Starting shutdown...
150831 12:08:22  InnoDB: Shutdown completed; log sequence number 0 44253
150831 12:08:22 [Note] /usr/libexec/mysqld: Shutdown complete

150831 12:08:22 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150831 12:08:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150831 12:08:46  InnoDB: Initializing buffer pool, size = 8.0M
150831 12:08:46  InnoDB: Completed initialization of buffer pool
150831 12:08:46  InnoDB: Started; log sequence number 0 44253
150831 12:08:46 [Note] Event Scheduler: Loaded 0 events
150831 12:08:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

running that alternate fix method didn't seem to fix it.

[tgriep]

Are you still having issues logging in to the XI GUI?

[snapon_admin]

No I can log into the GUI, it's just all broken.

brokenDB.png

[tgriep]

Lets stop the nagios processes and run the repair again.

Code: Select all

service nagios stop
service npcd stop
service nagios stop
killall -9 nagios
service cron stop
service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f *
service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
php /usr/local/nagiosxi/cron/dbmaint.php
service ndo2db start
service npcd start
service nagios start
service cron start

Let us know if this fixes it.

[jdalrymple]

I tried restoring from the most recent backup (yesterdays) and got this error at the end of the backup restore:

Did you change the username or password for the nod db? We install it with a specified known set of creds that are in the backup script. If you changed either of those you might have to spruce up the restore script to match. Also - are you sure it wasn't blown up at the time that you tried restoring from? The weekend gives a lot of time at which it could have failed.

[snapon_admin]

tgriep, no that doesn't appear to have fixed the issue. jdalrymple, I have not changed those passwords (at least not intentionally, I'm not even sure HOW to do that), and no I'm not sure that the backup from this weekend was before it blew up. It's actually quite likely to have been after if I had to guess. I have one other backup from last week that I can try.

[tgriep]

Before you try the restore, can you run the following and post back?

Code: Select all

df -h
df -i
ps aux
tail -50 /var/log/mysqld.log

[snapon_admin]

Code: Select all

[root@keno-ngos-01-pv nagios]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       97G   50G   42G  55% /
tmpfs                 3.9G     0  3.9G   0% /dev/shm
/dev/sda1             485M   50M  410M  11% /boot
[root@keno-ngos-01-pv nagios]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
                     6335472  158463 6177009    3% /
tmpfs                1006755       1 1006754    1% /dev/shm
/dev/sda1             128016      44  127972    1% /boot
[root@keno-ngos-01-pv nagios]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  19232  1424 ?        Ss   09:43   0:01 /sbin/init
root         2  0.0  0.0      0     0 ?        S    09:43   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/0]
root         4  0.0  0.0      0     0 ?        S    09:43   0:00 [ksoftirqd/0]
root         5  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/0]
root         6  0.0  0.0      0     0 ?        S    09:43   0:00 [watchdog/0]
root         7  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/1]
root         8  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/1]
root         9  0.0  0.0      0     0 ?        S    09:43   0:00 [ksoftirqd/1]
root        10  0.0  0.0      0     0 ?        S    09:43   0:00 [watchdog/1]
root        11  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/2]
root        12  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/2]
root        13  0.0  0.0      0     0 ?        S    09:43   0:00 [ksoftirqd/2]
root        14  0.0  0.0      0     0 ?        S    09:43   0:00 [watchdog/2]
root        15  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/3]
root        16  0.0  0.0      0     0 ?        S    09:43   0:00 [migration/3]
root        17  0.0  0.0      0     0 ?        S    09:43   0:00 [ksoftirqd/3]
root        18  0.0  0.0      0     0 ?        S    09:43   0:00 [watchdog/3]
root        19  0.0  0.0      0     0 ?        S    09:43   0:00 [events/0]
root        20  0.0  0.0      0     0 ?        S    09:43   0:00 [events/1]
root        21  0.0  0.0      0     0 ?        S    09:43   0:00 [events/2]
root        22  0.0  0.0      0     0 ?        S    09:43   0:00 [events/3]
root        23  0.0  0.0      0     0 ?        S    09:43   0:00 [cgroup]
root        24  0.0  0.0      0     0 ?        S    09:43   0:00 [khelper]
root        25  0.0  0.0      0     0 ?        S    09:43   0:00 [netns]
root        26  0.0  0.0      0     0 ?        S    09:43   0:00 [async/mgr]
root        27  0.0  0.0      0     0 ?        S    09:43   0:00 [pm]
root        28  0.0  0.0      0     0 ?        S    09:43   0:00 [sync_supers]
root        29  0.0  0.0      0     0 ?        S    09:43   0:00 [bdi-default]
root        30  0.0  0.0      0     0 ?        S    09:43   0:00 [kintegrityd/0]
root        31  0.0  0.0      0     0 ?        S    09:43   0:00 [kintegrityd/1]
root        32  0.0  0.0      0     0 ?        S    09:43   0:00 [kintegrityd/2]
root        33  0.0  0.0      0     0 ?        S    09:43   0:00 [kintegrityd/3]
root        34  0.0  0.0      0     0 ?        S    09:43   0:00 [kblockd/0]
root        35  0.0  0.0      0     0 ?        S    09:43   0:00 [kblockd/1]
root        36  0.0  0.0      0     0 ?        S    09:43   0:00 [kblockd/2]
root        37  0.0  0.0      0     0 ?        S    09:43   0:00 [kblockd/3]
root        38  0.0  0.0      0     0 ?        S    09:43   0:00 [kacpid]
root        39  0.0  0.0      0     0 ?        S    09:43   0:00 [kacpi_notify]
root        40  0.0  0.0      0     0 ?        S    09:43   0:00 [kacpi_hotplug]
root        41  0.0  0.0      0     0 ?        S    09:43   0:00 [ata/0]
root        42  0.0  0.0      0     0 ?        S    09:43   0:00 [ata/1]
root        43  0.0  0.0      0     0 ?        S    09:43   0:00 [ata/2]
root        44  0.0  0.0      0     0 ?        S    09:43   0:00 [ata/3]
root        45  0.0  0.0      0     0 ?        S    09:43   0:00 [ata_aux]
root        46  0.0  0.0      0     0 ?        S    09:43   0:00 [ksuspend_usbd]
root        47  0.0  0.0      0     0 ?        S    09:43   0:00 [khubd]
root        48  0.0  0.0      0     0 ?        S    09:43   0:00 [kseriod]
root        49  0.0  0.0      0     0 ?        S    09:43   0:00 [md/0]
root        50  0.0  0.0      0     0 ?        S    09:43   0:00 [md/1]
root        51  0.0  0.0      0     0 ?        S    09:43   0:00 [md/2]
root        52  0.0  0.0      0     0 ?        S    09:43   0:00 [md/3]
root        53  0.0  0.0      0     0 ?        S    09:43   0:00 [md_misc/0]
root        54  0.0  0.0      0     0 ?        S    09:43   0:00 [md_misc/1]
root        55  0.0  0.0      0     0 ?        S    09:43   0:00 [md_misc/2]
root        56  0.0  0.0      0     0 ?        S    09:43   0:00 [md_misc/3]
root        57  0.0  0.0      0     0 ?        S    09:43   0:00 [khungtaskd]
root        58  0.0  0.0      0     0 ?        S    09:43   0:08 [kswapd0]
root        59  0.0  0.0      0     0 ?        SN   09:43   0:00 [ksmd]
root        60  0.0  0.0      0     0 ?        SN   09:43   0:02 [khugepaged]
root        61  0.0  0.0      0     0 ?        S    09:43   0:00 [aio/0]
root        62  0.0  0.0      0     0 ?        S    09:43   0:00 [aio/1]
root        63  0.0  0.0      0     0 ?        S    09:43   0:00 [aio/2]
root        64  0.0  0.0      0     0 ?        S    09:43   0:00 [aio/3]
root        65  0.0  0.0      0     0 ?        S    09:43   0:00 [crypto/0]
root        66  0.0  0.0      0     0 ?        S    09:43   0:00 [crypto/1]
root        67  0.0  0.0      0     0 ?        S    09:43   0:00 [crypto/2]
root        68  0.0  0.0      0     0 ?        S    09:43   0:00 [crypto/3]
root        73  0.0  0.0      0     0 ?        S    09:43   0:00 [kthrotld/0]
root        74  0.0  0.0      0     0 ?        S    09:43   0:00 [kthrotld/1]
root        75  0.0  0.0      0     0 ?        S    09:43   0:00 [kthrotld/2]
root        76  0.0  0.0      0     0 ?        S    09:43   0:00 [kthrotld/3]
root        77  0.0  0.0      0     0 ?        S    09:43   0:00 [pciehpd]
root        79  0.0  0.0      0     0 ?        S    09:43   0:00 [kpsmoused]
root        80  0.0  0.0      0     0 ?        S    09:43   0:00 [usbhid_resumer]
root       110  0.0  0.0      0     0 ?        S    09:43   0:00 [kstriped]
root       292  0.0  0.0      0     0 ?        S    09:43   0:00 [scsi_eh_0]
root       293  0.0  0.0      0     0 ?        S    09:43   0:00 [scsi_eh_1]
root       305  0.0  0.0      0     0 ?        S    09:43   0:00 [mpt_poll_0]
root       306  0.0  0.0      0     0 ?        S    09:43   0:00 [mpt/0]
root       307  0.0  0.0      0     0 ?        S    09:43   0:00 [scsi_eh_2]
root       374  0.0  0.0      0     0 ?        S    09:43   0:00 [kdmflush]
root       376  0.0  0.0      0     0 ?        S    09:43   0:00 [kdmflush]
root       393  0.0  0.0      0     0 ?        S    09:43   0:03 [jbd2/dm-0-8]
root       394  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       395  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       396  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       397  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       470  0.0  0.0  10908  1004 ?        S<s  09:43   0:00 /sbin/udevd -d
root       670  0.0  0.0      0     0 ?        S    09:43   0:00 [vmmemctl]
root       705  0.0  0.0      0     0 ?        S    09:43   0:00 [hd-audio0]
root       793  0.0  0.0  10904  1036 ?        S<   09:43   0:00 /sbin/udevd -d
root       821  0.0  0.0      0     0 ?        S    09:43   0:00 [jbd2/sda1-8]
root       822  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       823  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       824  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       825  0.0  0.0      0     0 ?        S    09:43   0:00 [ext4-dio-unwrit]
root       857  0.0  0.0      0     0 ?        S    09:43   0:00 [kauditd]
root      1094  0.0  0.0  27664   848 ?        S<sl 09:43   0:00 auditd
root      1110  0.0  0.0 249084  3912 ?        Sl   09:43   0:01 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
dbus      1122  0.0  0.0  21404   804 ?        Ss   09:43   0:00 dbus-daemon --system
root      1159  0.0  0.0  64400  1204 ?        Ss   09:43   0:00 /usr/sbin/sshd
root      1167  0.0  0.0  22096   936 ?        Ss   09:43   0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
ntp       1175  0.0  0.0  30720  2020 ?        Ss   09:43   0:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
postgres  1350  0.0  0.0 214048  6180 ?        S    09:43   0:01 /usr/bin/postmaster -p 5432 -D /var/lib/pgsql/data
root      1353  0.2  0.0      0     0 ?        S    09:43   0:27 [flush-253:0]
postgres  1354  0.0  0.0 177060  1448 ?        Ss   09:43   0:00 postgres: logger process                          
postgres  1356  0.0  0.0 214168  6112 ?        Ss   09:43   0:01 postgres: writer process                          
postgres  1357  0.0  0.0 214048  1760 ?        Ss   09:43   0:01 postgres: wal writer process                      
postgres  1358  0.0  0.0 214184  1952 ?        Ss   09:43   0:00 postgres: autovacuum launcher process             
postgres  1359  0.0  0.0 177192  1676 ?        Ss   09:43   0:01 postgres: stats collector process                 
root      1437  0.0  0.0  79072  3364 ?        Ss   09:43   0:00 /usr/libexec/postfix/master
postfix   1446  0.0  0.0  79324  3240 ?        S    09:43   0:00 qmgr -l -t fifo -u
root      1447  0.0  0.1 337720 13832 ?        Ss   09:43   0:00 /usr/sbin/httpd
root      1455  0.0  0.0 117216  1244 ?        Ss   09:43   0:00 crond
ajaxterm  1472  0.0  0.0 168208  7712 ?        Sl   09:43   0:03 python /usr/share/ajaxterm/ajaxterm.py --daemon --port=8022 --uid=ajaxterm
apache    1493  2.4  0.4 462048 36636 ?        S    09:43   5:20 /usr/sbin/httpd
apache    1494  2.4  0.4 461700 36376 ?        S    09:43   5:17 /usr/sbin/httpd
apache    1495  2.4  0.4 461660 36292 ?        S    09:43   5:17 /usr/sbin/httpd
apache    1496  2.4  0.4 462204 36684 ?        S    09:43   5:18 /usr/sbin/httpd
apache    1497  2.4  0.4 461544 36156 ?        S    09:43   5:18 /usr/sbin/httpd
apache    1498  2.4  0.4 462204 36600 ?        S    09:43   5:18 /usr/sbin/httpd
apache    1499  2.4  0.4 461692 36264 ?        S    09:43   5:16 /usr/sbin/httpd
apache    1500  2.4  0.4 461708 36236 ?        S    09:43   5:18 /usr/sbin/httpd
postgres  1501  0.0  0.0 215964  7144 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53719) idle       
postgres  1502  0.0  0.0 215964  7596 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53720) idle       
postgres  1503  0.0  0.0 215920  7248 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53721) idle       
postgres  1504  0.0  0.0 215976  7112 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53722) idle       
root      1562  0.0  0.0   4064   568 tty1     Ss+  09:43   0:00 /sbin/mingetty /dev/tty1
root      1564  0.0  0.0   4064   568 tty2     Ss+  09:43   0:00 /sbin/mingetty /dev/tty2
root      1566  0.0  0.0  10904   988 ?        S<   09:43   0:00 /sbin/udevd -d
root      1567  0.0  0.0   4064   572 tty3     Ss+  09:43   0:00 /sbin/mingetty /dev/tty3
root      1569  0.0  0.0   4064   568 tty4     Ss+  09:43   0:00 /sbin/mingetty /dev/tty4
root      1571  0.0  0.0   4064   568 tty5     Ss+  09:43   0:00 /sbin/mingetty /dev/tty5
root      1573  0.0  0.0   4064   568 tty6     Ss+  09:43   0:00 /sbin/mingetty /dev/tty6
apache    1574  2.4  0.4 461544 36200 ?        S    09:43   5:17 /usr/sbin/httpd
postgres  1575  0.0  0.0 215968  6892 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53726) idle       
postgres  1578  0.0  0.0 216016  6960 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53727) idle       
postgres  1579  0.0  0.0 216016  6892 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53728) idle       
postgres  1581  0.0  0.0 216016  7452 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53729) idle       
postgres  1595  0.0  0.0 215968  6856 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53731) idle       
apache    1610  2.4  0.4 464364 36936 ?        S    09:43   5:15 /usr/sbin/httpd
apache    1611  2.4  0.4 461656 36328 ?        S    09:43   5:20 /usr/sbin/httpd
postgres  1612  0.0  0.0 216060  7652 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53736) idle       
postgres  1615  0.0  0.0 215972  7124 ?        Ss   09:43   0:07 postgres: nagiosxi nagiosxi ::1(53737) idle       
apache    4953  2.4  0.4 461588 36196 ?        S    09:45   5:16 /usr/sbin/httpd
postgres  4960  0.0  0.0 215452  6820 ?        Ss   09:45   0:07 postgres: nagiosxi nagiosxi ::1(55496) idle       
postfix   7745  0.0  0.0  79152  3332 ?        S    13:03   0:00 pickup -l -t fifo -u
apache   17212  2.3  0.4 460936 35564 ?        S    09:55   4:53 /usr/sbin/httpd
postgres 17240  0.0  0.0 215972  6964 ?        Ss   09:55   0:07 postgres: nagiosxi nagiosxi ::1(35477) idle       
apache   20094  2.3  0.4 461192 35520 ?        S    09:58   4:49 /usr/sbin/httpd
postgres 20120  0.0  0.0 215968  6852 ?        Ss   09:58   0:07 postgres: nagiosxi nagiosxi ::1(37343) idle       
apache   22537  2.3  0.4 462056 36668 ?        S    10:00   4:46 /usr/sbin/httpd
postgres 22574  0.0  0.0 216008  7392 ?        Ss   10:00   0:07 postgres: nagiosxi nagiosxi ::1(38985) idle       
root     26991  0.0  0.0 140100  1728 ?        S    13:20   0:00 CROND
root     26992  0.0  0.0 140100  1728 ?        S    13:20   0:00 CROND
root     26993  0.0  0.0 140100  1728 ?        S    13:20   0:00 CROND
root     26994  0.0  0.0 140100  1728 ?        S    13:20   0:00 CROND
root     26995  0.0  0.0 140100  1728 ?        S    13:20   0:00 CROND
nagios   26999  0.0  0.0   9196  1188 ?        Ss   13:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios   27000  0.0  0.0   9196  1188 ?        Ss   13:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios   27006  0.0  0.0   9196  1188 ?        Ss   13:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios   27007  0.0  0.0   9196  1188 ?        Ss   13:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios   27008  0.0  0.0   9196  1184 ?        Ss   13:20   0:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios   27009  0.4  0.3 232500 30280 ?        S    13:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php
nagios   27012  0.3  0.2 224984 22804 ?        S    13:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios   27014  0.3  0.2 224812 22296 ?        S    13:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios   27015  0.5  0.3 231976 25360 ?        S    13:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios   27017  0.3  0.2 224688 22452 ?        S    13:20   0:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
postgres 27022  0.0  0.0 215456  6224 ?        Ss   13:20   0:00 postgres: nagiosxi nagiosxi ::1(56539) idle       
postgres 27030  0.0  0.0 215420  5896 ?        Ss   13:20   0:00 postgres: nagiosxi nagiosxi ::1(56543) idle       
postgres 27031  0.0  0.0 215420  5924 ?        Ss   13:20   0:00 postgres: nagiosxi nagiosxi ::1(56544) idle       
postgres 27038  0.0  0.0 215420  5928 ?        Ss   13:20   0:00 postgres: nagiosxi nagiosxi ::1(56546) idle       
postgres 27045  0.0  0.0 215456  6536 ?        Ss   13:20   0:00 postgres: nagiosxi nagiosxi ::1(56548) idle       
nagios   28024  0.0  0.0  41420  2884 ?        S    13:20   0:00 /usr/local/nagios/libexec/check_nrpe -H kentest11g.snapon.com -u -p 5668 -t 70 -c check_zone_cpu -a -n kenamem01t -w 80 -
nagios   28033  0.0  0.0  41420  2884 ?        S    13:20   0:00 /usr/local/nagios/libexec/check_nrpe -H kenamem01d.snapon.com -u -p 5668 -t 70 -c check_init_service -a svc:/network/inet
root     28034  0.0  0.0 110236  1172 pts/0    R+   13:20   0:00 ps aux
root     28505  0.0  0.0  98160  4024 ?        Ss   10:05   0:02 sshd: root@pts/0 
root     28609  0.0  0.0 108452  1972 pts/0    Ss   10:05   0:00 -bash
apache   49930  2.4  0.4 462844 37496 ?        S    10:26   4:17 /usr/sbin/httpd
postgres 49939  0.0  0.0 215972  6780 ?        Ss   10:26   0:06 postgres: nagiosxi nagiosxi ::1(57028) idle       
root     61820  0.0  0.0 108168  1564 pts/0    S    12:54   0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysql
mysql    61922  5.1  0.5 2229452 47336 pts/0   Sl   12:54   1:22 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/
nagios   62245  0.0  0.0  50192   848 ?        Ss   12:55   0:00 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   62256  0.0  0.0 368888  1040 ?        S    12:55   0:00 /usr/local/nagios/bin/npcd -d -f /usr/local/nagios/etc/pnp/npcd.cfg
nagios   62294  1.6  0.2  37348 16568 ?        Ss   12:55   0:24 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   62296  0.1  0.0  10016   952 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62297  0.0  0.0  10016   952 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62298  0.0  0.0  10016   952 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62299  0.0  0.0  10016   948 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62300  0.0  0.0  10016   960 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62301  0.0  0.0  10016   952 ?        S    12:55   0:01 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   62302  0.1  0.0  50192  1592 ?        S    12:55   0:01 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   62303  1.0  0.1  61240 12756 ?        S    12:55   0:16 /usr/local/nagios/bin/ndo2db -c /usr/local/nagios/etc/ndo2db.cfg
nagios   62326  0.0  0.1  36832 11220 ?        S    12:55   0:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
[root@keno-ngos-01-pv nagios]# tail -50 /var/log/mysqld.log
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:07:43 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:01 [ERROR] /usr/libexec/mysqld: Incorrect key file for table './nagios/nagios_servicestatus.MYI'; try to repair it
150831 12:08:18 [Note] /usr/libexec/mysqld: Normal shutdown

150831 12:08:18 [Note] Event Scheduler: Purging the queue. 0 events
150831 12:08:20  InnoDB: Starting shutdown...
150831 12:08:22  InnoDB: Shutdown completed; log sequence number 0 44253
150831 12:08:22 [Note] /usr/libexec/mysqld: Shutdown complete

150831 12:08:22 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150831 12:08:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150831 12:08:46  InnoDB: Initializing buffer pool, size = 8.0M
150831 12:08:46  InnoDB: Completed initialization of buffer pool
150831 12:08:46  InnoDB: Started; log sequence number 0 44253
150831 12:08:46 [Note] Event Scheduler: Loaded 0 events
150831 12:08:46 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
150831 12:50:20 [Note] /usr/libexec/mysqld: Normal shutdown

150831 12:50:20 [Note] Event Scheduler: Purging the queue. 0 events
150831 12:50:22  InnoDB: Starting shutdown...
150831 12:50:24  InnoDB: Shutdown completed; log sequence number 0 44253
150831 12:50:24 [Note] /usr/libexec/mysqld: Shutdown complete

150831 12:50:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150831 12:54:05 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150831 12:54:05  InnoDB: Initializing buffer pool, size = 8.0M
150831 12:54:05  InnoDB: Completed initialization of buffer pool
150831 12:54:05  InnoDB: Started; log sequence number 0 44253
150831 12:54:05 [Note] Event Scheduler: Loaded 0 events
150831 12:54:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.71'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution