Page 3 of 4

Re: Service checks keep postponing

Posted: Wed Jun 10, 2015 2:22 pm
by tgriep
Could you run the following command and post the output here?

Code: Select all

ll /usr/local/nagios/var/

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 11:10 am
by rkymtnhigh
Sure!

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 11:51 am
by jdalrymple
Are there any ugly messages in /var/log/mysqld.log?

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 12:06 pm
by rkymtnhigh
jdalrymple wrote:Are there any ugly messages in /var/log/mysqld.log?
I think we are getting somewhere. I see tons of "Incorrect key file for table XXX - try to repair it"

Seems to be servicestatus and systemcommands mainly. I did run a table repair before posting, but maybe I need to get more aggressive?

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 12:10 pm
by abrist
Yeah, try to run our full database repair script:

Code: Select all

/usr/local/nagiosxi/scripts/repair_databases.sh
Once complete, wait 10 minutes, check the last check time, and then finally check the mysqld logs again:

Code: Select all

tail -25 /var/log/mysqld.log

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 12:33 pm
by rkymtnhigh
I'm not seeing any more table errors, in fact I havent gotten any events since I repaired the tables.
Checks still are not running :cry:

Re: Service checks keep postponing

Posted: Thu Jun 11, 2015 1:28 pm
by jdalrymple

Code: Select all

/etc/init.d/nagios stop
/etc/init.d/ndo2db restart
/etc/init.d/nagios start
or maybe even

Code: Select all

reboot

Re: Service checks keep postponing

Posted: Fri Jun 12, 2015 10:45 am
by rkymtnhigh
no luck. after rebooting it forced a check which completed successfully (except for those still in pending), but they did not run again.

I really appreciate all the help, this just doesn't look good :(

Re: Service checks keep postponing

Posted: Fri Jun 12, 2015 11:23 am
by jdalrymple
It's salvageable, always is. Is mysqld.log clean right now? Please look again.

Re: Service checks keep postponing

Posted: Fri Jun 12, 2015 11:30 am
by rkymtnhigh
Hardly any events since the reboot yesterday