Page 3 of 3

Re: check_http is stuck in pending.

Posted: Thu Jun 28, 2012 9:49 am
by scottwilkerson
I've been trying to replicate this (unsuccessfully), can you tell me when you add a new check_http service are you using a wizard, doing it through the CCM or other?

Re: check_http is stuck in pending.

Posted: Mon Jul 02, 2012 1:19 am
by s.dehaan
scottwilkerson wrote:I've been trying to replicate this (unsuccessfully), can you tell me when you add a new check_http service are you using a wizard, doing it through the CCM or other?
I have been using the CCM.

Re: check_http is stuck in pending.

Posted: Thu Jul 05, 2012 12:01 pm
by scottwilkerson
Can I have you try running the standard website wizard for one to see if they are working properly.

Thanks

Re: check_http is stuck in pending.

Posted: Thu Jul 05, 2012 12:15 pm
by s.dehaan
scottwilkerson wrote:Can I have you try running the standard website wizard for one to see if they are working properly.

Thanks
That's kind of a problem because I removed the services associated with the Nagios XI wizzard... Is there a way to get them back without reinstalling?

Or when I reinstall, is there a way to import my current settings without removing existing things?


:oops:

Re: check_http is stuck in pending.

Posted: Thu Jul 05, 2012 1:10 pm
by scottwilkerson
You can restore the default commands and templates by doing the following, unzip attached file and copy xiwzardtemplates.cfg to
/usr/local/nagios/etc/import

run the following from the command line

Code: Select all

chown nagios:nagios /usr/local/nagios/etc/import/xiwzardtemplates.cfg
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh

Re: check_http is stuck in pending.

Posted: Fri Jul 06, 2012 2:42 am
by lmilkovic
We had similar issues on several occasions and they were all related to "corrupt" NDO database.

Some hosts and services were shown in PENDING state, services would have notifications disabled at random etc.
Applying new configuration from the CCM would not help, since the historic data in NDO database is still present.

This issue was very hard to debug and I haven't really figured out what was the cause.
However, I did find some correlations - it occurred more often when upgrading to a new version of Nagios XI, when multiple Nagios servers are writing to a single NDO database (extremely uncommon setup and done for highly specific purposes) or when the MySQL database is replicated between multiple nodes.

Regarding the solution, I first tried to manually search through the database (http://nagios.sourceforge.net/docs/ndou ... _Model.pdf is a necessary resource here) identifying and deleting "suspicious" entries. You should search both historic and current tables.
It's difficult to tell what entries are suspicious in general, though:(

After some time I found it too tiresome and I usually fixed this by restoring a recent database backup...

I know this is not a direct solution, but I hope you find at least some information here useful:)

Re: check_http is stuck in pending.

Posted: Fri Jul 06, 2012 9:00 am
by scottwilkerson
lmilkovic wrote:We had similar issues on several occasions and they were all related to "corrupt" NDO database.

Some hosts and services were shown in PENDING state, services would have notifications disabled at random etc.
Applying new configuration from the CCM would not help, since the historic data in NDO database is still present.

This issue was very hard to debug and I haven't really figured out what was the cause.
However, I did find some correlations - it occurred more often when upgrading to a new version of Nagios XI, when multiple Nagios servers are writing to a single NDO database (extremely uncommon setup and done for highly specific purposes) or when the MySQL database is replicated between multiple nodes.

Regarding the solution, I first tried to manually search through the database (http://nagios.sourceforge.net/docs/ndou ... _Model.pdf is a necessary resource here) identifying and deleting "suspicious" entries. You should search both historic and current tables.
It's difficult to tell what entries are suspicious in general, though:(

After some time I found it too tiresome and I usually fixed this by restoring a recent database backup...

I know this is not a direct solution, but I hope you find at least some information here useful:)
this is what I was thinking too and why I suggested repairing the DB back here
http://support.nagios.com/forum/viewtop ... 211#p28092

Re: check_http is stuck in pending.

Posted: Sun Jul 08, 2012 4:29 pm
by lmilkovic
Sorry Scott, I haven't seen that post:(

However, in my experience, the database was not corrupted in "MySQL-style-corruption", i.e. all tables were functional, no indexes were corrupt etc. The database structure and internals were working fine, but some "ghost" entries and other suspicious entries were poisoning the database and causing the issues similar to the one described here. I had to manually delete those entries (for example: entries for the same check and same check time, but containing different results/textual descriptions).

It is ofcourse recommended to run the repairmysql script as well, as you advised.

[SOLVED]check_http is stuck in pending.

Posted: Mon Jul 09, 2012 1:45 am
by s.dehaan
scottwilkerson wrote:You can restore the default commands and templates by doing the following, unzip attached file and copy xiwzardtemplates.cfg to
/usr/local/nagios/etc/import

run the following from the command line

Code: Select all

chown nagios:nagios /usr/local/nagios/etc/import/xiwzardtemplates.cfg
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
Turns out that using the NagiosXI wizzard to create this services fixes the problem. No explanation why... But it works!

Re: check_http is stuck in pending.

Posted: Mon Jul 09, 2012 7:52 am
by scottwilkerson
Excellent! :D