mysqld: Table is crashed and last (automatic?) repair

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: mysqld: Table is crashed and last (automatic?) repair

Post by tgriep »

Thanks for the files.
What I see is that the MYSQL database was stopped and started while the nagios and the ndo2db services were running and that causes corruption to the database.

If you ever have to do any maintenance to the MYSQL database like restarting it, stop the nagios, ndo2db, crond services first and that should stop the corruption from happening.

The notifications table is so corrupted, you cannot describe the table settings so we will have to truncate it.

Run the following to stop the processes, repair and truncate the database and then start the process again.

Code: Select all

systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop crond
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
echo "truncate table nagios_notifications;" | mysql -u root -pnagiosxi nagios
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
mysqlcheck -f -o -u root -pnagiosxi --databases nagios
systemctl restart mariadb
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
rm -f /usr/local/nagiosxi/tmp/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill python
systemctl restart httpd
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

As soon as you are done with the above, run this as root and post the output here so I can see if the notifications table settings look correct.

Code: Select all

echo 'desc nagios_notifications;' |mysql -t -u root -pnagiosxi nagios
Be sure to check out our Knowledgebase for helpful articles and solutions!
capgemtools
Posts: 56
Joined: Wed Aug 12, 2020 3:16 pm

Re: mysqld: Table is crashed and last (automatic?) repair

Post by capgemtools »

Hello Tgriep,

I ran the commands in given sequence the output file of commands run is attached(
command_output.log
command_output.log).

I see following error and the original error still continues in mariadb logs

nagios.nagios_notifications
error : Can't create new tempfile: '/var/lib/mysql/nagios/nagios_notifications.TMM'
status : Operation failed

[root@pngoawg001 tmp]# echo 'desc nagios_notifications;' |mysql -t -u root -pnagiosxi nagios
ERROR 144 (HY000) at line 1: Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed
You do not have the required permissions to view the files attached to this post.
Thanks,
Supriya
capgemtools
Posts: 56
Joined: Wed Aug 12, 2020 3:16 pm

Re: mysqld: Table is crashed and last (automatic?) repair

Post by capgemtools »

Just checked the file exists with 345mb size.

[root@pngoawg001 tmp]# ls -l /var/lib/mysql/nagios/nagios_notifications*
-rw-rw---- 1 mysql mysql 9168 Aug 5 04:37 /var/lib/mysql/nagios/nagios_notifications.frm
-rw-rw---- 1 mysql mysql 0 Dec 17 12:10 /var/lib/mysql/nagios/nagios_notifications.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 17 12:10 /var/lib/mysql/nagios/nagios_notifications.MYI
-rw-rw---- 1 mysql mysql 361441280 Aug 5 05:41 /var/lib/mysql/nagios/nagios_notifications.TMM

[root@pngoawg001 tmp]# du -sh /var/lib/mysql/nagios/nagios_notifications.TMM
345M /var/lib/mysql/nagios/nagios_notifications.TMM
Thanks,
Supriya
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: mysqld: Table is crashed and last (automatic?) repair

Post by tgriep »

I answered in the PM but I'll put the same answer here.
Yes, the nagios_notifications.TMM file is causing the repair command from repairing the notifications table so delete it

Code: Select all

rm -f /var/lib/mysql/nagios/nagios_notifications.TMM
Run the repair of the table and continue on with the process.
Be sure to check out our Knowledgebase for helpful articles and solutions!
capgemtools
Posts: 56
Joined: Wed Aug 12, 2020 3:16 pm

Re: mysqld: Table is crashed and last (automatic?) repair

Post by capgemtools »

Hi Tgriep,

Attached is the table settings.

Let me know any additional steps need to be performed.

For now I am monitoring the mariadb logs for any errors. if anything found will report here.

Thanks for your help on this.
You do not have the required permissions to view the files attached to this post.
Thanks,
Supriya
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: mysqld: Table is crashed and last (automatic?) repair

Post by tgriep »

The table settings look good so you should be good to go.
Be sure to check out our Knowledgebase for helpful articles and solutions!
capgemtools
Posts: 56
Joined: Wed Aug 12, 2020 3:16 pm

Re: mysqld: Table is crashed and last (automatic?) repair

Post by capgemtools »

Thanks. I do not see any more errors. Its fixed now.
Thanks,
Supriya
User avatar
tgriep
Madmin
Posts: 9190
Joined: Thu Oct 30, 2014 9:02 am

Re: mysqld: Table is crashed and last (automatic?) repair

Post by tgriep »

Your Welcome. Glad the issue is fixed now. I'll close the post for you but feel free to open a new one in the future for any questions or issues.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Locked