Page 1 of 2
mysql_error Crashed table entries in /var/log/messages
Posted: Thu Jan 11, 2018 1:19 pm
by bsivavani
Hi,
We are seeing crashed table entries in /var/log/messages, but in the front end Nagios XI 5.2.9 console everything looks working and config applying successfully.
Do we need to repair the DB ? Kindly let us know what is the future impact if not do DB Repair?
Kindly advice?
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Thu Jan 11, 2018 2:56 pm
by npolovenko
Hello,
@bsivavani.
It could be something insignificant. Perhaps the server was shutdown incorrectly at some point in past? Or maybe was running out of memory? I'd recommend to backup XI (Admin/Local backups), so in case something goes wrong you could revert and restore all configs.
Any good reason why you wouldn't want to run a database repair script?
https://support.nagios.com/kb/article.php?id=24
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Tue Jan 16, 2018 8:16 am
by bsivavani
We are using DRBD to run Nagios. While repairing DB, Nagios will get restart automatically and get fail because it is running under DRBD.
Can you let us know how to repair DB when Nagios services are running under DRBD setup ?
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Tue Jan 16, 2018 1:09 pm
by npolovenko
@bsivavani, Generally speaking, we don't support any custom configurations, including DRBD. If running the repair script is not an option try this command:
Code: Select all
mysqlcheck -r -f -uroot -pnagiosxi --all-databases
Or if you can find the name of the crashed database table in the messages file, you can try to repair just that table:
Code: Select all
echo 'repair table <broken table> use_frm;' | mysql -t -u root -pnagiosxi nagios
echo 'repair table <broken table> use_frm;' | mysql -t -u root -pnagiosxi nagiosql
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Tue Jan 16, 2018 11:14 pm
by bsivavani
Do we need to stop mysql db before running these commands?
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Wed Jan 17, 2018 4:57 am
by bsivavani
Yes. MySQL should be in start state before running below commands
echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagiosql
Please find attachment for Repair table output. Please let us know if this is expected output ?
We are not seeing errors now. We will monitor it for couple of days.
Also we have observed that in sysctl.conf file kernel.msgmni set to 2878. Do we need to increase this value to avoid future msg queue full issues. Kindly confirm ?
[root@XXXXX log]# grep 'kernel.msgmni' /etc/sysctl.conf
kernel.msgmni = 2878
[root@XXXXX log]#
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Wed Jan 17, 2018 11:03 am
by npolovenko
@bsivavani, Sounds good. Yes, let us know if more Database errors will show up.
As far as kernel.msgmni, yes, I do recommend to increase that. We have an article that will walk you through:
https://support.nagios.com/kb/article.php?id=139
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Fri Jan 19, 2018 6:26 am
by bsivavani
Hi,
We are not seeing any mysql erros in /var/log/mesages, but we have observed that local backups are not happening.
We see below message in Scheduled backups tab, but backups are not storing under /store/backups/nagiosxi. Last backup is on Jan 10th.
Next Run: Sat, Jan 20th at 5:00AM (+01:00 CET)
Last Run: Fri, Jan 19th at 5:00AM (+01:00 CET)
Kindly advice ?
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Fri Jan 19, 2018 10:17 am
by npolovenko
@bsivavani, Please navigate to:
And run the following script:
And then run a backup script manually and let us know if it shows any errors:
Re: mysql_error Crashed table entries in /var/log/messages
Posted: Wed Feb 07, 2018 9:20 am
by bsivavani
Hi,
We tried to take manual backup using backup_xi.sh script, but we are seeing below errors.
all nagios services are in stopped state.
[root@XXXX ~]# /usr/local/nagiosxi/scripts/backup_xi.sh
Running configuration check...
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.s ock' (2) when trying to connect
Error backing up MySQL database 'nagios' - check the password in this script!
[root@XXXX ~]#
Kindly advice on this error ?