Checks stop running randomly

[KevinD]

Alas... no change...

Ran the DB upgrade (had to make some minor tweaks due to primary keys cols(varchar) being changed to text cols (which doesn't work in MySQL)) but nothing major.

The count on the frozen checks continues to rise until we trigger the script to force a check.

I would LOVE to do a remote session.
Feel free to email, PM, USPS, FedEX, smoke signals, or telepathically transmit that and we can get something arranged for 2 minutes from now if your available.

[mguthrie]

Just a follow up for other users who may read. This issue appears to be caused be using DNX with the Nagios Core 3.4.1 engine. Not sure of the exact cause yet. More detail will be posted as we find out.

[mguthrie]

Hey what version of the Nagios Core was running on the DNX slave machine(s)?

[gwakem]

Those were upgraded from r2.3 to r3.1 also, so Nagios Core 3.4.1. They have no configuration, so their databases are local and we didn't need to modify to the upgrade scripts.

[mguthrie]

Did my best to try and hunt this one down, but tracing it through the event timing loop is pretty brutal. I actually passed the question onto the nagios-devel mailing list on sourceforge to see if they have any ideas.

[gwakem]

We appreciate your work and help on tracking it down. I'm going to start investigating mod gearman on Monday, since it seems to be highly recommended. Any pointers or docs that have proven helpful for gottchas would be appreciated, but I'll be scouring the page as well as the documents section next week. Have a great weekend guys.

[scottwilkerson]

Mod-gearman has the required items in RPM form now at http://mod-gearman.org/download/ so the install looks something like this

Code: Select all

cd /tmp
wget http://mod-gearman.org/download/v1.3.0/rhel6/x86_64/gearmand-0.25-1.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v1.3.0/rhel6/x86_64/gearmand-devel-0.25-1.rhel6.x86_64.rpm
wget http://mod-gearman.org/download/v1.3.0/rhel6/x86_64/mod_gearman-1.3.0-1.e.rhel6.x86_64.rpm
yum --nogpgcheck -y install gearmand-0.25-1.rhel6.x86_64.rpm
yum --nogpgcheck -y install gearmand-devel-0.25-1.rhel6.x86_64.rpm
yum --nogpgcheck -y install mod_gearman-1.3.0-1.e.rhel6.x86_64.rpm

you will need to add the NEB to the nagios.cfg, something like this

Code: Select all

broker_module=/usr/local/share/nagios/mod_gearman.o keyfile=/usr/local/share/nagios/secret.txt server=localhost eventhandler=yes hosts=yes services=yes

On each of the workers we need to adjust the config at /etc/mod_gearman/mod_gearman_worker.conf
workers we need to adjust the following values
server address is that of your gearmand server (usually Nagios Server)
server=localhost:4730
Shared key or keyfile needs to be the same on both the workers and gearmand server
key=should_be_changed
or
keyfile=

[mguthrie]

We were finally able to located and correct the check issue with DNX. The fix will be available in the upcoming 3.3 release, but if you'd like a patch sooner than that let me know and I'll get that to you. (Not sure if you moved to Mod Gearman or not).

[gwakem]

We did move to mod_gearman, and it is actually filling our needs a bit better. Thank you for letting us know though!