Page 2 of 2

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Thu Dec 17, 2020 10:14 am
by tgriep
Thanks for the files.
What I see is that the MYSQL database was stopped and started while the nagios and the ndo2db services were running and that causes corruption to the database.

If you ever have to do any maintenance to the MYSQL database like restarting it, stop the nagios, ndo2db, crond services first and that should stop the corruption from happening.

The notifications table is so corrupted, you cannot describe the table settings so we will have to truncate it.

Run the following to stop the processes, repair and truncate the database and then start the process again.

Code: Select all

systemctl stop npcd
systemctl stop nagios
systemctl stop ndo2db
systemctl stop crond
pkill -9 -u nagios
echo "truncate table xi_events; truncate table xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi
echo "truncate table nagios_notifications;" | mysql -u root -pnagiosxi nagios
mysqlcheck -f -r -u root -pnagiosxi --all-databases --use-frm
mysqlcheck -f -o -u root -pnagiosxi --databases nagios
systemctl restart mariadb
rm -f /usr/local/nagios/var/rw/nagios.cmd
rm -f /usr/local/nagios/var/nagios.lock
rm -f /var/run/nagios.lock
rm -f /usr/local/nagios/var/ndo.sock
rm -f /usr/local/nagios/var/ndo2db.lock
rm -f /var/lib/mrtg/mrtg_l
rm -f /usr/local/nagiosxi/var/*.lock
rm -f /usr/local/nagiosxi/tmp/*.lock
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
pkill python
systemctl restart httpd
systemctl start ndo2db
systemctl start nagios
systemctl start npcd
systemctl start crond

As soon as you are done with the above, run this as root and post the output here so I can see if the notifications table settings look correct.

Code: Select all

echo 'desc nagios_notifications;' |mysql -t -u root -pnagiosxi nagios

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Thu Dec 17, 2020 1:24 pm
by capgemtools
Hello Tgriep,

I ran the commands in given sequence the output file of commands run is attached(
command_output.log
command_output.log).

I see following error and the original error still continues in mariadb logs

nagios.nagios_notifications
error : Can't create new tempfile: '/var/lib/mysql/nagios/nagios_notifications.TMM'
status : Operation failed

[root@pngoawg001 tmp]# echo 'desc nagios_notifications;' |mysql -t -u root -pnagiosxi nagios
ERROR 144 (HY000) at line 1: Table './nagios/nagios_notifications' is marked as crashed and last (automatic?) repair failed

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Thu Dec 17, 2020 1:29 pm
by capgemtools
Just checked the file exists with 345mb size.

[root@pngoawg001 tmp]# ls -l /var/lib/mysql/nagios/nagios_notifications*
-rw-rw---- 1 mysql mysql 9168 Aug 5 04:37 /var/lib/mysql/nagios/nagios_notifications.frm
-rw-rw---- 1 mysql mysql 0 Dec 17 12:10 /var/lib/mysql/nagios/nagios_notifications.MYD
-rw-rw---- 1 mysql mysql 1024 Dec 17 12:10 /var/lib/mysql/nagios/nagios_notifications.MYI
-rw-rw---- 1 mysql mysql 361441280 Aug 5 05:41 /var/lib/mysql/nagios/nagios_notifications.TMM

[root@pngoawg001 tmp]# du -sh /var/lib/mysql/nagios/nagios_notifications.TMM
345M /var/lib/mysql/nagios/nagios_notifications.TMM

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Thu Dec 17, 2020 6:06 pm
by tgriep
I answered in the PM but I'll put the same answer here.
Yes, the nagios_notifications.TMM file is causing the repair command from repairing the notifications table so delete it

Code: Select all

rm -f /var/lib/mysql/nagios/nagios_notifications.TMM
Run the repair of the table and continue on with the process.

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Fri Dec 18, 2020 9:25 am
by capgemtools
Hi Tgriep,

Attached is the table settings.

Let me know any additional steps need to be performed.

For now I am monitoring the mariadb logs for any errors. if anything found will report here.

Thanks for your help on this.

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Fri Dec 18, 2020 10:16 am
by tgriep
The table settings look good so you should be good to go.

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Tue Jan 05, 2021 8:14 am
by capgemtools
Thanks. I do not see any more errors. Its fixed now.

Re: mysqld: Table is crashed and last (automatic?) repair

Posted: Tue Jan 05, 2021 10:12 am
by tgriep
Your Welcome. Glad the issue is fixed now. I'll close the post for you but feel free to open a new one in the future for any questions or issues.