mysql_error Crashed table entries in /var/log/messages

[bsivavani]

Hi,

We are seeing crashed table entries in /var/log/messages, but in the front end Nagios XI 5.2.9 console everything looks working and config applying successfully.

Do we need to repair the DB ? Kindly let us know what is the future impact if not do DB Repair?

Kindly advice?

[npolovenko]

Hello, @bsivavani.
It could be something insignificant. Perhaps the server was shutdown incorrectly at some point in past? Or maybe was running out of memory? I'd recommend to backup XI (Admin/Local backups), so in case something goes wrong you could revert and restore all configs.
Any good reason why you wouldn't want to run a database repair script?
https://support.nagios.com/kb/article.php?id=24

[bsivavani]

We are using DRBD to run Nagios. While repairing DB, Nagios will get restart automatically and get fail because it is running under DRBD.

Can you let us know how to repair DB when Nagios services are running under DRBD setup ?

[npolovenko]

@bsivavani, Generally speaking, we don't support any custom configurations, including DRBD. If running the repair script is not an option try this command:

Code: Select all

 mysqlcheck -r -f -uroot -pnagiosxi --all-databases

Or if you can find the name of the crashed database table in the messages file, you can try to repair just that table:

Code: Select all

echo 'repair table <broken table> use_frm;' | mysql -t -u root -pnagiosxi nagios
echo 'repair table <broken table> use_frm;' | mysql -t -u root -pnagiosxi nagiosql

[bsivavani]

Do we need to stop mysql db before running these commands?

[bsivavani]

Yes. MySQL should be in start state before running below commands

echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagios
echo 'repair table nagios_logentries use_frm;' | mysql -t -u root -pnagiosxi nagiosql

Please find attachment for Repair table output. Please let us know if this is expected output ?

We are not seeing errors now. We will monitor it for couple of days.

Also we have observed that in sysctl.conf file kernel.msgmni set to 2878. Do we need to increase this value to avoid future msg queue full issues. Kindly confirm ?

[root@XXXXX log]# grep 'kernel.msgmni' /etc/sysctl.conf
kernel.msgmni = 2878
[root@XXXXX log]#

[npolovenko]

@bsivavani, Sounds good. Yes, let us know if more Database errors will show up.
As far as kernel.msgmni, yes, I do recommend to increase that. We have an article that will walk you through:
https://support.nagios.com/kb/article.php?id=139

[bsivavani]

Hi,

We are not seeing any mysql erros in /var/log/mesages, but we have observed that local backups are not happening.

We see below message in Scheduled backups tab, but backups are not storing under /store/backups/nagiosxi. Last backup is on Jan 10th.
Next Run: Sat, Jan 20th at 5:00AM (+01:00 CET)
Last Run: Fri, Jan 19th at 5:00AM (+01:00 CET)

Kindly advice ?

[npolovenko]

@bsivavani, Please navigate to:

Code: Select all

/usr/local/nagiosxi/scripts/

And run the following script:

Code: Select all

./reset_config_perms.sh

And then run a backup script manually and let us know if it shows any errors:

Code: Select all

./backup_xi.sh

[bsivavani]

Hi,

We tried to take manual backup using backup_xi.sh script, but we are seeing below errors.

all nagios services are in stopped state.

[root@XXXX ~]# /usr/local/nagiosxi/scripts/backup_xi.sh
Running configuration check...
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.s ock' (2) when trying to connect
Error backing up MySQL database 'nagios' - check the password in this script!
[root@XXXX ~]#

Kindly advice on this error ?