mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
Posted: Sun Jan 12, 2025 7:19 am
Hello,
It has been noticed that 'Monitoring Engine' in Nagios XI has been failing often recently. On checking the mariadb logs, it says that DB tables are crashed and should be repaired.
Mariadb logs:
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_programstatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_programstatus'
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_hoststatus'
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_servicestatus'
250110 21:56:48 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and should be repaired
250110 21:56:48 [Warning] Checking table: './nagios/nagios_logentries'
250110 21:56:49 [ERROR] mysqld: Table './nagios/nagios_comments' is marked as crashed and should be repaired
250110 21:56:49 [Warning] Checking table: './nagios/nagios_comments'
250110 21:58:01 [ERROR] mysqld: Table './nagiosxi/xi_eventqueue' is marked as crashed and should be repaired
250110 21:58:01 [Warning] Checking table: './nagiosxi/xi_eventqueue'
250110 22:00:01 [ERROR] mysqld: Table './nagios/nagios_commenthistory' is marked as crashed and should be repaired
250110 22:00:01 [Warning] Checking table: './nagios/nagios_commenthistory'
250110 22:00:02 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and should be repaired
250110 22:00:02 [Warning] Checking table: './nagios/nagios_notifications'
250110 22:00:02 [ERROR] mysqld: Table './nagios/nagios_contactnotifications' is marked as crashed and should be repaired
250110 22:00:02 [Warning] Checking table: './nagios/nagios_contactnotifications'
250110 22:00:03 [ERROR] mysqld: Table './nagios/nagios_contactnotificationmethods' is marked as crashed and should be repaired
250110 22:00:03 [Warning] Checking table: './nagios/nagios_contactnotificationmethods'
250110 22:00:03 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
250110 22:00:03 [Warning] Checking table: './nagios/nagios_statehistory'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_systemcommands'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_servicechecks' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_servicechecks'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_hostchecks' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_hostchecks'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_eventhandlers' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_eventhandlers'
250110 22:12:33 [ERROR] mysqld: Table './nagios/nagios_flappinghistory' is marked as crashed and should be repaired
250110 22:12:33 [Warning] Checking table: './nagios/nagios_flappinghistory'
As a workaround, we stop crond, npcd, nagios, nagiosxi, ndo2db, mariadb and httpd services, repair DB by running '/usr/local/nagiosxi/scripts/repair_databases.sh' command and start the services again.
It stays healthy for some time until 'Monitoring Engine' fails again. We are looking for permanent fix as the issue has been occurring again and again.
It has been noticed that 'Monitoring Engine' in Nagios XI has been failing often recently. On checking the mariadb logs, it says that DB tables are crashed and should be repaired.
Mariadb logs:
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_programstatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_programstatus'
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_hoststatus'
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
250110 21:56:47 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
250110 21:56:47 [Warning] Checking table: './nagios/nagios_servicestatus'
250110 21:56:48 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and should be repaired
250110 21:56:48 [Warning] Checking table: './nagios/nagios_logentries'
250110 21:56:49 [ERROR] mysqld: Table './nagios/nagios_comments' is marked as crashed and should be repaired
250110 21:56:49 [Warning] Checking table: './nagios/nagios_comments'
250110 21:58:01 [ERROR] mysqld: Table './nagiosxi/xi_eventqueue' is marked as crashed and should be repaired
250110 21:58:01 [Warning] Checking table: './nagiosxi/xi_eventqueue'
250110 22:00:01 [ERROR] mysqld: Table './nagios/nagios_commenthistory' is marked as crashed and should be repaired
250110 22:00:01 [Warning] Checking table: './nagios/nagios_commenthistory'
250110 22:00:02 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and should be repaired
250110 22:00:02 [Warning] Checking table: './nagios/nagios_notifications'
250110 22:00:02 [ERROR] mysqld: Table './nagios/nagios_contactnotifications' is marked as crashed and should be repaired
250110 22:00:02 [Warning] Checking table: './nagios/nagios_contactnotifications'
250110 22:00:03 [ERROR] mysqld: Table './nagios/nagios_contactnotificationmethods' is marked as crashed and should be repaired
250110 22:00:03 [Warning] Checking table: './nagios/nagios_contactnotificationmethods'
250110 22:00:03 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
250110 22:00:03 [Warning] Checking table: './nagios/nagios_statehistory'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_systemcommands'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_servicechecks' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_servicechecks'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_hostchecks' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_hostchecks'
250110 22:00:09 [ERROR] mysqld: Table './nagios/nagios_eventhandlers' is marked as crashed and should be repaired
250110 22:00:09 [Warning] Checking table: './nagios/nagios_eventhandlers'
250110 22:12:33 [ERROR] mysqld: Table './nagios/nagios_flappinghistory' is marked as crashed and should be repaired
250110 22:12:33 [Warning] Checking table: './nagios/nagios_flappinghistory'
As a workaround, we stop crond, npcd, nagios, nagiosxi, ndo2db, mariadb and httpd services, repair DB by running '/usr/local/nagiosxi/scripts/repair_databases.sh' command and start the services again.
It stays healthy for some time until 'Monitoring Engine' fails again. We are looking for permanent fix as the issue has been occurring again and again.