Nagios DB repair failed... now what?

[lynrob]

I am new to Nagios XI and still learning the ropes. Today I decided I would reboot my virtual machine to make sure I understood how that worked. Unfortunately, something must have gone wrong because when I brought Nagios back up I was getting lots of Nagios system errors. I noticed that the mysqld daemon was taking nearly 100% of the CPU so I checked logs and found that mysql was complaining about 3 corrupted tables. I followed the instructions in the http://assets.nagios.com/downloads/nagi ... tabase.pdf document and still got the following error when running /usr/local/nagiosxi/cron/dbmaint.php:

SQL: DELETE FROM nagios_hostchecks WHERE start_time < FROM_UNIXTIME(1317944940)
SQL: SQL Error [ndoutils] :</b> Table './nagios/nagios_hostchecks' is marked as crashed and last (automatic?) repair failedCLEANING ndoutils TABLE 'eventhandlers'...

The document said to contact support if a similar error was encountered so that is the purpose of this email. I am not sure what to do next. The document says it might be necessary to truncate some files but I'm not sure how I would recover the lost data.

Also, is there a way to properly shutdown the VM without clobbering the DB? Thanks!!

[lmiltchev]

You may need to run a force repair on the tables:

service mysqld stop
cd /var/lib/mysql/nagios
myisamchk -r -f nagios_<corrupted_table>

service mysqld start
rm -f /usr/local/nagiosxi/var/dbmaint.lock
/usr/local/nagiosxi/cron/dbmaint.php

Check our wiki page on the topic here:

http://support.nagios.com/wiki/index.ph ... s_Stale.22

I will add these steps to the "Nagios XI - Repairing The Database" document; it has to be updated.

Hope this helps. Let us know if you have any more issues.

[lynrob]

Thanks for your quick reply... The steps you mentioned are already in the "Nagios XI - Repairing The Database" document and they worked for 2 of the 3 corrupted tables. For the remaining table, I was able to "fix" it by getting into mysql and running "repair table nagios_hostchecks;" . Now I'm no longer getting errors in the mysql error log and Nagios seems to be running fine now.

Before I powered down my VM, I stopped Nagios but in retrospect, I think that I should have stopped mysql as well. Are there other daemons that I should have shut down prior to powering down the VM?

Thanks again for you help. You guys have been very patient with this newbie and very helpful.

[lmiltchev]

I believe stopping mysql should be sufficient. I'm glad your problem got solved. Thank you for the feedback!