SQL Problems I think

[[email protected]]

We had a forced shutdown of the NagiosXI server and after reboot I get the following error under "Tactical Overview":

SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL Error [ndoutils] : Incorrect file format 'nagios_hoststatus' SQL: SQL

I ran the "Nagios XI – Repairing The Database" commands but still having same issue.

Please let me know what we can do. NagiosXI is down.

[jdalrymple]

Sometimes it's helpful to run the repair script more than once.

Please post a current (meaning after you ran the repair script) tail of /var/log/mysqld.log.

[[email protected]]

Ran the SQL fix 3 times and tailed the log. Anything unusual?

===============
REPAIR COMPLETE
===============
[root@localhost tmp]# service mysqld start
Starting mysqld: [ OK ]
[root@localhost tmp]# tail -50 /var/log/mysqld.log
150712 10:54:19 InnoDB: Completed initialization of buffer pool
150712 10:54:19 InnoDB: Started; log sequence number 0 44233
150712 10:54:19 [Note] Event Scheduler: Loaded 0 events
150712 10:54:19 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150713 14:10:40 [Note] /usr/libexec/mysqld: Normal shutdown

150713 14:10:40 [Note] Event Scheduler: Purging the queue. 0 events
150713 14:10:40 InnoDB: Starting shutdown...
150713 14:10:45 InnoDB: Shutdown completed; log sequence number 0 44233
150713 14:10:45 [Note] /usr/libexec/mysqld: Shutdown complete

150713 14:10:45 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150713 14:10:54 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150713 14:10:54 InnoDB: Initializing buffer pool, size = 8.0M
150713 14:10:54 InnoDB: Completed initialization of buffer pool
150713 14:10:54 InnoDB: Started; log sequence number 0 44233
150713 14:10:54 [Note] Event Scheduler: Loaded 0 events
150713 14:10:54 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150713 14:11:07 [Note] /usr/libexec/mysqld: Normal shutdown

150713 14:11:07 [Note] Event Scheduler: Purging the queue. 0 events
150713 14:11:07 InnoDB: Starting shutdown...
150713 14:11:10 InnoDB: Shutdown completed; log sequence number 0 44233
150713 14:11:10 [Note] /usr/libexec/mysqld: Shutdown complete

150713 14:11:10 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150713 14:11:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150713 14:11:11 InnoDB: Initializing buffer pool, size = 8.0M
150713 14:11:11 InnoDB: Completed initialization of buffer pool
150713 14:11:11 InnoDB: Started; log sequence number 0 44233
150713 14:11:11 [Note] Event Scheduler: Loaded 0 events
150713 14:11:11 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
150713 14:11:14 [Note] /usr/libexec/mysqld: Normal shutdown

150713 14:11:14 [Note] Event Scheduler: Purging the queue. 0 events
150713 14:11:14 InnoDB: Starting shutdown...
150713 14:11:16 InnoDB: Shutdown completed; log sequence number 0 44233
150713 14:11:16 [Note] /usr/libexec/mysqld: Shutdown complete

150713 14:11:16 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
150713 14:11:17 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150713 14:11:17 InnoDB: Initializing buffer pool, size = 8.0M
150713 14:11:17 InnoDB: Completed initialization of buffer pool
150713 14:11:17 InnoDB: Started; log sequence number 0 44233
150713 14:11:17 [Note] Event Scheduler: Loaded 0 events
150713 14:11:17 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
[root@localhost tmp]#

[jdalrymple]

Yeah, nothing bad in the log. Are you still seeing the error in the UI?

If so it might be advisable to just restore the system from a known good backup. At this point you're not monitoring anything I don't think anyway. What does the tail of /usr/local/nagios/var/nagios.log look like?

[[email protected]]

Still get the same errors in GUI. Here is the /usr/local/nagios/var/nagios.log

I don't think I have a good backup... Not sure where they reside. I have upgraded a couple times and the messages showed that it was backing up before the upgrade.

[1436810467] SERVICE ALERT: phxsalb.grhc.local;/opt Disk Space;OK;SOFT;2;DISK OK - free space: /opt 222 MB (31% inode=83%):
[1436812287] SERVICE ALERT: phxsa.grhc.local;Total Processes;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436812339] SERVICE ALERT: phxsa.grhc.local;Total Processes;OK;SOFT;2;PROCS OK: 1421 processes
[1436812499] Auto-save of retention data completed successfully.
[1436813297] SERVICE ALERT: phxsa.grhc.local;/usr disk space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436813349] SERVICE ALERT: phxsa.grhc.local;/usr disk space;OK;SOFT;2;DISK OK - free space: /usr 1943 MB (37% inode=89%):
[1436815968] SERVICE ALERT: phxsaqm3.grhc.local;Jrn_shadow Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436816019] SERVICE ALERT: phxsaqm3.grhc.local;Jrn_shadow Disk Space;OK;SOFT;2;DISK OK - free space: /jrn_shadow 37302 MB (80% inode=99%):
[1436816099] Auto-save of retention data completed successfully.
[1436816423] SERVICE ALERT: phxsa3.grhc.local;/tmp Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436816424] SERVICE ALERT: phxsatsm1.grhc.local;/TSM/ddpl2 Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436816476] SERVICE ALERT: phxsa3.grhc.local;/tmp Disk Space;OK;SOFT;2;DISK OK - free space: /tmp 2440 MB (79% inode=99%):
[1436816477] SERVICE ALERT: phxsatsm1.grhc.local;/TSM/ddpl2 Disk Space;OK;SOFT;2;DISK OK - free space: /TSM/ddpl2 534957 MB (4% inode=99%):
[1436816646] SERVICE ALERT: phxsatsm1.grhc.local;/TSM/archlog Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436816699] SERVICE ALERT: phxsatsm1.grhc.local;/TSM/archlog Disk Space;OK;SOFT;2;DISK OK - free space: /TSM/archlog 482555 MB (85% inode=99%):
[1436818184] SERVICE ALERT: phxsaqm3.grhc.local;Total Processes;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436818234] SERVICE ALERT: phxsaqm3.grhc.local;Total Processes;OK;SOFT;2;PROCS OK: 132 processes
[1436818383] SERVICE ALERT: phxsatsm1.grhc.local;/var Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436818407] SERVICE ALERT: phxsaqm3.grhc.local;User Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436818436] SERVICE ALERT: phxsatsm1.grhc.local;/var Disk Space;OK;SOFT;2;DISK OK - free space: /var 1099 MB (26% inode=95%):
[1436818460] SERVICE ALERT: phxsaqm3.grhc.local;User Disk Space;OK;SOFT;2;DISK OK - free space: /usr 2319 MB (27% inode=85%):
[1436819698] Auto-save of retention data completed successfully.
[1436821147] SERVICE ALERT: phxsalb.grhc.local;Root Disk Space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436821199] SERVICE ALERT: phxsalb.grhc.local;Root Disk Space;OK;SOFT;2;DISK OK - free space: / 1816 MB (85% inode=97%):
[1436821700] SERVICE ALERT: phxsa.grhc.local;/usr disk space;CRITICAL;SOFT;1;CRITICAL - Plugin timed out after 10 seconds
[1436821751] SERVICE ALERT: phxsa.grhc.local;/usr disk space;OK;SOFT;2;DISK OK - free space: /usr 1943 MB (37% inode=89%):
[1436823298] Auto-save of retention data completed successfully.

[[email protected]]

I found the backup directory, restored a month old back and we are good to go...

Thanks

[tgriep]

That is good to hear. Can we close this post for you?