Table is marked as crashed in production

[bsivavani]

Hi,

We have observed that in production environment we are receiving below errors. We did repaired the database but still we are seeing the errors in /var/log/mysqld.log file

180417 16:22:39 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:22:45 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:22 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:39 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:43 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:23:44 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:37 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:41 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:24:43 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:25:40 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:25:42 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:26:41 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:28:07 [ERROR] /usr/libexec/mysqld: Table './nagsnt/tbl_sn_ticket_tracker_test' is marked as crashed and should be repaired
180417 16:29:01 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:01 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:29:03 [ERROR] /usr/libexec/mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
180417 16:34:21 [ERROR] Got error 127 when reading table './nagiosxi/xi_sysstat'

Kindly advice the next steps as it is effecting the production monitoring?

[scottwilkerson]

Here is our doc on repairing database, you will likely also need to repair the nagsnt database which is separate from the doc
https://assets.nagios.com/downloads/nag ... tabase.pdf

[bsivavani]

We did already tried to repair db using below pdf but no luck.
https://assets.nagios.com/downloads/nag ... tabase.pdf

Kindly advice on how to repair below tables
nagiosxi/xi_meta
nagiosxi/xi_events
nagsnt/tbl_sn_ticket_tracker_test

[scottwilkerson]

should be able to run this:

Code: Select all

service mysqld stop
./repairmysql.sh nagiosxi xi_meta
./repairmysql.sh nagiosxi xi_events
./repairmysql.sh nagsnt tbl_sn_ticket_tracker_test
service mysqld start

If that doesn't work see page 4 of the doc for alternate repair methods

[manojkonda]

Hi scottwilkerson,

Thanks for the reply,

We have also observed that in the CCM Tab, the Data is blank nothing is getting reflected.
Is this because of the tables getting crashed. ? Any advise on restoring the CCM data ?

[scottwilkerson]

Can you see the items when you go to the links on the left
Hosts
Services
Contact, etc?

[manojkonda]

Nope, it doesn't display anything when clicked on the respective links.

[scottwilkerson]

This sounds like you have something not working in the nagiosql database.

If the DB's cannot be repaired, I would strongly suggest restoring to a backup if you have a recent one.

[bsivavani]

DB got repaired. We are not seeing any errors in /var/log/mysqld.log and the current monitoring also working and incidents also creating.

The Only issue is with config data not populating in Nagios console. Kindly let us know how we can get the configs under CCM ?

[scottwilkerson]

Go to CCM -> Configuration Snapshots
Click the restore icon next to the top icon
Then OK
Then you can verify if the items show up under hosts,services, etc

If so, all should be good, if not let us know, and we may need a closer look