Page 1 of 2

Table is marked as crashed in production

Posted: Tue Apr 17, 2018 9:37 am
by bsivavani
Hi,

We have observed that in production environment we are receiving below errors. We did repaired the database but still we are seeing the errors in /var/log/mysqld.log file

180417 16:22:39 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:22:45 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:39 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:43 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:44 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:37 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:41 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:43 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:25:40 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:25:42 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:26:41 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:29:01 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:01 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:34:21 [ERROR] Got error 127 when reading table './nagiosxi/xi_sysstat'

Kindly advice the next steps as it is effecting the production monitoring?

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 10:02 am
by scottwilkerson
Here is our doc on repairing database, you will likely also need to repair the nagsnt database which is separate from the doc
https://assets.nagios.com/downloads/nag ... tabase.pdf

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 10:09 am
by bsivavani
We did already tried to repair db using below pdf but no luck.
https://assets.nagios.com/downloads/nag ... tabase.pdf

Kindly advice on how to repair below tables
nagiosxi/xi_meta
nagiosxi/xi_events
nagsnt/tbl_sn_ticket_tracker_test

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 10:25 am
by scottwilkerson
should be able to run this:

Code: Select all

service mysqld stop
./repairmysql.sh nagiosxi xi_meta
./repairmysql.sh nagiosxi xi_events
./repairmysql.sh nagsnt tbl_sn_ticket_tracker_test
service mysqld start
If that doesn't work see page 4 of the doc for alternate repair methods

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 10:45 am
by manojkonda
Hi scottwilkerson,

Thanks for the reply,

We have also observed that in the CCM Tab, the Data is blank nothing is getting reflected.
Is this because of the tables getting crashed. ? Any advise on restoring the CCM data ?

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 11:03 am
by scottwilkerson
Can you see the items when you go to the links on the left
Hosts
Services
Contact, etc?

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 11:09 am
by manojkonda
Nope, it doesn't display anything when clicked on the respective links.

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 11:19 am
by scottwilkerson
This sounds like you have something not working in the nagiosql database.

If the DB's cannot be repaired, I would strongly suggest restoring to a backup if you have a recent one.

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 12:38 pm
by bsivavani
DB got repaired. We are not seeing any errors in /var/log/mysqld.log and the current monitoring also working and incidents also creating.

The Only issue is with config data not populating in Nagios console. Kindly let us know how we can get the configs under CCM ?

Re: Table is marked as crashed in production

Posted: Tue Apr 17, 2018 12:46 pm
by scottwilkerson
Go to CCM -> Configuration Snapshots
Click the restore icon next to the top icon
Then OK
Then you can verify if the items show up under hosts,services, etc

If so, all should be good, if not let us know, and we may need a closer look