Page 1 of 1

Table error in DB

Posted: Mon Oct 03, 2016 9:13 am
by fodorm
I am seeing this error repeated in the message logs:

Oct 2 03:26:05 horcrux2 ndo2db: mysql_error: 'Table './nagios/nagios_logentries' is marked as crashed and last (automatic?) repair failed'

We had an incident over the weekend where Nagios was alerting numerous items down but spot checks showed the items up. Any ideas?

Thanks,

-Matt

Re: Table error in DB

Posted: Mon Oct 03, 2016 10:55 am
by lmiltchev
Run the DB repair script:

Code: Select all

cd /usr/local/nagiosxi/scripts
./repair_databases.sh
restart mysqld:

Code: Select all

service mysqld restart
and check the log for errors/crashed tables:

Code: Select all

tail -50 /var/log/mysqld.log

Re: Table error in DB

Posted: Mon Oct 03, 2016 12:05 pm
by fodorm
I ran the script and am seeing this when tailing the log:

161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:07 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
161003 13:04:08 [ERROR] /usr/libexec/mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired

Re: Table error in DB

Posted: Mon Oct 03, 2016 12:21 pm
by lmiltchev
How much space to you have on the hard drive?

Code: Select all

df -h
Try running:

Code: Select all

mysqlcheck -r -f -uroot -pnagiosxi --all-databases
restart mysqld:

Code: Select all

service mysqld restart
and check the log again:

Code: Select all

tail /var/log/mysqld.log

Re: Table error in DB

Posted: Mon Oct 03, 2016 12:28 pm
by fodorm
The issue now appears resolved. Thanks for your assistance.

Re: Table error in DB

Posted: Mon Oct 03, 2016 12:44 pm
by avandemore
Glad this resolved your issue fodorm. I'm closing this thread now. Please open a new one for any further issues.