Storage crash and database errors
Posted: Mon Oct 17, 2011 6:03 pm
We had a major storage crash (good times let me assure you) last week and it appears to have corrupted the Nagios XI database. The application still appears to be running ok but /var/log/messages is filled with these fun messages:
Oct 18 09:52:07 hostname ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_id='1', logentry_time=FROM_UNIXTIME(1318891926), entry_time=FROM_UNIXTIME(1318891926), entry_time_usec='908347', logentry_type='64', logentry_data='Finished daemonizing\.\.\. \(New PID=11235\)', realtime_data='1', inferred_data_extracted='1''
Oct 18 09:52:07 hostname ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed'
Oct 18 09:53:07 hostname ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_eventhandlers WHERE instance_id='1' AND start_time<FROM_UNIXTIME(1316213587)'
Oct 18 09:53:07 hostname ndo2db: mysql_error: 'Table './nagios/nagios_eventhandlers' is marked as crashed and last (automatic?) repair failed'
Oct 18 09:54:08 hostname ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_eventhandlers WHERE instance_id='1' AND start_time<FROM_UNIXTIME(1316213648)'
Oct 18 09:54:08 hostname ndo2db: mysql_error: 'Table './nagios/nagios_eventhandlers' is marked as crashed and last (automatic?) repair failed'
I'm using the standard 32-bit CentOS VMware image, is there a database health script hidden somewhere that might help restore order?
Oct 18 09:52:07 hostname ndo2db: Error: mysql_query() failed for 'INSERT INTO nagios_logentries SET instance_id='1', logentry_time=FROM_UNIXTIME(1318891926), entry_time=FROM_UNIXTIME(1318891926), entry_time_usec='908347', logentry_type='64', logentry_data='Finished daemonizing\.\.\. \(New PID=11235\)', realtime_data='1', inferred_data_extracted='1''
Oct 18 09:52:07 hostname ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed'
Oct 18 09:53:07 hostname ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_eventhandlers WHERE instance_id='1' AND start_time<FROM_UNIXTIME(1316213587)'
Oct 18 09:53:07 hostname ndo2db: mysql_error: 'Table './nagios/nagios_eventhandlers' is marked as crashed and last (automatic?) repair failed'
Oct 18 09:54:08 hostname ndo2db: Error: mysql_query() failed for 'DELETE FROM nagios_eventhandlers WHERE instance_id='1' AND start_time<FROM_UNIXTIME(1316213648)'
Oct 18 09:54:08 hostname ndo2db: mysql_error: 'Table './nagios/nagios_eventhandlers' is marked as crashed and last (automatic?) repair failed'
I'm using the standard 32-bit CentOS VMware image, is there a database health script hidden somewhere that might help restore order?