Nagios XI mysqld crashes

[madhudeva]

Hi Team,

We are facing frequent crashes of mysqld? can you suggest to prevent these?

we had to run these commnads to fix the error and alternately we had a crontab which runs every saturday.
on every weekend.#Repairing MySQL Tables
5 5 * * sat /sbin/service mysqld stop
8 5 * * sat /usr/local/nagiosxi/scripts/repairmysql.sh nagios
11 5 * * sat /sbin/service mysqld start


Regards,
Madhu

[slansing]

Database maintenance is already cronned in NagiosXI. Are you restarting your nagios server when you see these issues? IF so, how are you doing it?

[madhudeva]

we would run below commands to fix it..
Do we need to restart nagios after running below commands?

1. service mysqld stop
2. /usr/local/nagiosxi/scripts/repairmysql.sh nagios
3. service mysqld start

And i see below cronjobs for nagios xi. Which cron is for repairing database?
ps -ef | grep cron
nagios 1524 1452 0 07:00 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 1544 1524 0 07:00 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
root 2369 1 0 2013 ? 00:05:36 crond
nagios 21023 21009 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 21024 21020 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 21029 21021 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 21030 21023 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 21031 21008 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 21032 21024 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 21035 21010 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 21037 21029 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 21041 21031 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 21042 21035 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php


the below file is increasing and taking lots of space. Can you suggest what we can do?

/var/lib/mysql/nagios# ls -lh nagios_logentries.MYD
-rw-rw---- 1 mysql mysql 7.3G Mar 11 07:36 nagios_logentries.MYD

[slansing]

You should be running both of the below if manually fixing crashed tables via the script:

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql

Which as stated should already be done by XI on a cron. The script also stops and starts mysqld by itself so you do not need to manually do that.

[tmcdonald]

Do you have state stalking enabled for either hosts or services? That could explain the increasing log entries.

[madhudeva]

No.. we dont have any state stalking enabled.
Can we nullify the file? will it cause any problems?

[slansing]

Which file? And please note my reply above, as that may be one of your issues.

[madhudeva]

The below file I wanted to nullify.. or do we have any commands to truncate the logentries?

/var/lib/mysql/nagios# ls -lh nagios_logentries.MYD
-rw-rw---- 1 mysql mysql 7.3G Mar 11 07:36 nagios_logentries.MYD

And also please help me in few things.
We are restarting nagios everyday morning at 4 AM
Can we also restart mysqld everyday along with nagios say at 4:05 AM? Is it mandatory to restart nagios service after restarting/repairing mysqld?

# To Restart Nagios XI
0 4 * * * /sbin/service nagios restart

[sreinhardt]

You should be truncating if you wish to clear those. Please note, that this will effect some historical data within the xi interface, and potentially some state history reports. With that said, the repairing the nagios xi database document, has the two truncation commands you will want to run to get the most effect, then repair both databases one more time.

[madhudeva]

I have truncated log files which is around 9 GB , extenral commands table ( aroung 5 GB )and now nagios XI is running fine without any crashes.. and the server load is also normal..
Thank you for your help