Page 1 of 2

Reloading the XI config hangs

Posted: Mon Apr 15, 2013 11:49 am
by emarmonti
Greetings,

I recently migrated a Nagios XI installation from a Xen VM to a physical machine. It worked perfectly on the VM, but I'm having two issues that I think may be related to one another:

- When trying to reload the config, I get "Waiting for configuration verification....................." and it times out, failing to load the config.
- The Quick Find no longer auto-populates the field.


According to http://support.nagios.com/wiki/index.php/Nagios_XI:FAQs, the first problem could be a bug (if I was running 2011R1.3), but this is the latest version of XI:

- According to Admin / Check For Updates, I'm running version 2012R1.7.
- I've modified /etc/php.ini meets the minimum recommendations according to the above referenced FAQ:

max_execution_time = 75
max_input_time = 75
memory_limit = 512M

- Even though host lookups work fine, I've created entries for the hostname in /etc/hosts (we access it via nagios.domain.net, while the actual hostname is nagios2.domain.net).
- There is plenty of free disk space
- I've restarted httpd, I've rebooted multiple times.

But the problem persists.
Any recommendations of troubleshooting steps I might take?
I'm debating on downgrading to the previous version just to see if it behaves the same.




1.What version of Nagios XI are you using? - 2012R1.7
2.Linux Distribution and version? - CentOS 6.4
3.32 or 64bit? - 64Bit
4.VMware Image or Manual Install of XI? - Manual install
5.Are there specials configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL? - No


Thanks
EJ

Re: Reloading the XI config hangs

Posted: Mon Apr 15, 2013 1:09 pm
by abrist
Are you using IE? If so, you may have to enable "compatability mode".

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 11:36 am
by emarmonti
abrist wrote:Are you using IE? If so, you may have to enable "compatability mode".
I'm not using IE. This worked just fine in Firefox or Chrome on the previous installation.

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 12:40 pm
by scottwilkerson
Can we verify the cmdsubsys cron is running

Code: Select all

ps -ef |grep cmdsubsys
if so, can you tail that log while trying to apply configuration

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 1:53 pm
by emarmonti
scottwilkerson, the cmdsubsys is running and unfortunately the log says nothing indicating why this is happening as far as I can tell.


Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.
RESETTING PERMS
SETUID ROOT OK
/usr/local/nagiosxi/nom/checkpoints/nagioscore /usr/local/nagiosxi/scripts
tar: Removing leading `/' from member names
/usr/local/nagiosxi/scripts
OUTPUT=/usr/local/nagiosxi/scripts
RETURNCODE=0
.......................
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........................................................





Additionally, in the web gui, if I go to Monitoring Process/Process Info, I see "Monitoring Engine Process.... No data... Last Updated: <foo>". Where normally I should be able to control enabling/disable notifications, etc from there.


Thanks
ej


scottwilkerson wrote:Can we verify the cmdsubsys cron is running

Code: Select all

ps -ef |grep cmdsubsys
if so, can you tail that log while trying to apply configuration

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 2:01 pm
by scottwilkerson
emarmonti wrote:Additionally, in the web gui, if I go to Monitoring Process/Process Info, I see "Monitoring Engine Process.... No data... Last Updated: <foo>". Where normally I should be able to control enabling/disable notifications, etc from there.
Can we run the following from the command line

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start
Then go back and look at this page, if the info doesn't populate we may have a db problem

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 2:20 pm
by emarmonti
scottwilkerson wrote: Can we run the following from the command line

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start
Then go back and look at this page, if the info doesn't populate we may have a db problem
After following your instructions, there's still no information on the Process Info page. Is there a a tool built into nagios that validates core database integrity? What else would you suggest?


Thanks

Re: Reloading the XI config hangs

Posted: Tue Apr 16, 2013 3:59 pm
by abrist
Follow the steps in the document below to verify and repair the mysql database:
http://assets.nagios.com/downloads/nagi ... tabase.pdf

Re: Reloading the XI config hangs

Posted: Thu Apr 18, 2013 7:51 am
by emarmonti
I repaired the database per documentation. It finished within 30 seconds. mysqld started up fine with no errors after the repairs.

Unfortunately I'm still experiencing the same behavior. To summarize the issues again:

* The monitor engine process shows "no data" under Process Info
* When scheduling immediate service checks, I get ""Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy." (though the checks do in fact get scheduled immediately per cmdsubsys.log, XI does not recognize it).
* When applying configuration changes, I get "Waiting for configuration verification....................." and it times out. Though it does reload the config as I realized after looking at cmdsubsys.log, Nagios XI does not recognize it reloaded the config.
* The Quick Find no longer auto-populates the field as I start typing in a hostname.

It seems like Nagios XI is not properly communicating with Nagios core.

This is very frustrating, especially since I followed the documentation when I restored the configuration and it worked perfect before this. Any other ideas?



Thanks
abrist wrote:Follow the steps in the document below to verify and repair the mysql database:
http://assets.nagios.com/downloads/nagi ... tabase.pdf

Re: Reloading the XI config hangs

Posted: Thu Apr 18, 2013 12:48 pm
by scottwilkerson
Is crond running?

Code: Select all

service crond status
Also can you port the output of

Code: Select all

tail -20 /var/log/cron