Page 1 of 2

Nagios XI mysqld crashes

Posted: Mon Mar 10, 2014 1:46 am
by madhudeva
Hi Team,

We are facing frequent crashes of mysqld? can you suggest to prevent these?

we had to run these commnads to fix the error and alternately we had a crontab which runs every saturday.
on every weekend.#Repairing MySQL Tables
5 5 * * sat /sbin/service mysqld stop
8 5 * * sat /usr/local/nagiosxi/scripts/repairmysql.sh nagios
11 5 * * sat /sbin/service mysqld start


Regards,
Madhu

Re: Nagios XI mysqld crashes

Posted: Mon Mar 10, 2014 9:43 am
by slansing
Database maintenance is already cronned in NagiosXI. Are you restarting your nagios server when you see these issues? IF so, how are you doing it?

Re: Nagios XI mysqld crashes

Posted: Tue Mar 11, 2014 1:17 am
by madhudeva
we would run below commands to fix it..
Do we need to restart nagios after running below commands?

1. service mysqld stop
2. /usr/local/nagiosxi/scripts/repairmysql.sh nagios
3. service mysqld start

And i see below cronjobs for nagios xi. Which cron is for repairing database?
ps -ef | grep cron
nagios 1524 1452 0 07:00 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
nagios 1544 1524 0 07:00 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php
root 2369 1 0 2013 ? 00:05:36 crond
nagios 21023 21009 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
nagios 21024 21020 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
nagios 21029 21021 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
nagios 21030 21023 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php
nagios 21031 21008 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
nagios 21032 21024 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
nagios 21035 21010 0 07:12 ? 00:00:00 /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
nagios 21037 21029 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php
nagios 21041 21031 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php
nagios 21042 21035 0 07:12 ? 00:00:00 /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php


the below file is increasing and taking lots of space. Can you suggest what we can do?

/var/lib/mysql/nagios# ls -lh nagios_logentries.MYD
-rw-rw---- 1 mysql mysql 7.3G Mar 11 07:36 nagios_logentries.MYD

Re: Nagios XI mysqld crashes

Posted: Tue Mar 11, 2014 10:43 am
by slansing
You should be running both of the below if manually fixing crashed tables via the script:

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagios

Code: Select all

/usr/local/nagiosxi/scripts/repairmysql.sh nagiosql
Which as stated should already be done by XI on a cron. The script also stops and starts mysqld by itself so you do not need to manually do that.

Re: Nagios XI mysqld crashes

Posted: Tue Mar 11, 2014 10:45 am
by tmcdonald
Do you have state stalking enabled for either hosts or services? That could explain the increasing log entries.

Re: Nagios XI mysqld crashes

Posted: Wed Mar 12, 2014 7:56 am
by madhudeva
No.. we dont have any state stalking enabled.
Can we nullify the file? will it cause any problems?

Re: Nagios XI mysqld crashes

Posted: Wed Mar 12, 2014 2:06 pm
by slansing
Which file? And please note my reply above, as that may be one of your issues.

Re: Nagios XI mysqld crashes

Posted: Thu Mar 13, 2014 4:30 am
by madhudeva
The below file I wanted to nullify.. or do we have any commands to truncate the logentries?

/var/lib/mysql/nagios# ls -lh nagios_logentries.MYD
-rw-rw---- 1 mysql mysql 7.3G Mar 11 07:36 nagios_logentries.MYD

And also please help me in few things.
We are restarting nagios everyday morning at 4 AM
Can we also restart mysqld everyday along with nagios say at 4:05 AM? Is it mandatory to restart nagios service after restarting/repairing mysqld?

# To Restart Nagios XI
0 4 * * * /sbin/service nagios restart

Re: Nagios XI mysqld crashes

Posted: Thu Mar 13, 2014 10:55 am
by sreinhardt
You should be truncating if you wish to clear those. Please note, that this will effect some historical data within the xi interface, and potentially some state history reports. With that said, the repairing the nagios xi database document, has the two truncation commands you will want to run to get the most effect, then repair both databases one more time.

Re: Nagios XI mysqld crashes

Posted: Thu Apr 03, 2014 12:40 am
by madhudeva
I have truncated log files which is around 9 GB , extenral commands table ( aroung 5 GB )and now nagios XI is running fine without any crashes.. and the server load is also normal..
Thank you for your help :D