Reloading the XI config hangs

[emarmonti]

Greetings,

I recently migrated a Nagios XI installation from a Xen VM to a physical machine. It worked perfectly on the VM, but I'm having two issues that I think may be related to one another:

- When trying to reload the config, I get "Waiting for configuration verification....................." and it times out, failing to load the config.
- The Quick Find no longer auto-populates the field.


According to http://support.nagios.com/wiki/index.php/Nagios_XI:FAQs, the first problem could be a bug (if I was running 2011R1.3), but this is the latest version of XI:

- According to Admin / Check For Updates, I'm running version 2012R1.7.
- I've modified /etc/php.ini meets the minimum recommendations according to the above referenced FAQ:

max_execution_time = 75
max_input_time = 75
memory_limit = 512M

- Even though host lookups work fine, I've created entries for the hostname in /etc/hosts (we access it via nagios.domain.net, while the actual hostname is nagios2.domain.net).
- There is plenty of free disk space
- I've restarted httpd, I've rebooted multiple times.

But the problem persists.
Any recommendations of troubleshooting steps I might take?
I'm debating on downgrading to the previous version just to see if it behaves the same.




1.What version of Nagios XI are you using? - 2012R1.7
2.Linux Distribution and version? - CentOS 6.4
3.32 or 64bit? - 64Bit
4.VMware Image or Manual Install of XI? - Manual install
5.Are there specials configurations on your system, ie; is Gnome installed? Are you using a proxy? Are you using SSL? - No


Thanks
EJ

[abrist]

Are you using IE? If so, you may have to enable "compatability mode".

[emarmonti]

Are you using IE? If so, you may have to enable "compatability mode".

I'm not using IE. This worked just fine in Firefox or Chrome on the previous installation.

[scottwilkerson]

Can we verify the cmdsubsys cron is running

Code: Select all

ps -ef |grep cmdsubsys

if so, can you tail that log while trying to apply configuration

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[emarmonti]

scottwilkerson, the cmdsubsys is running and unfortunately the log says nothing indicating why this is happening as far as I can tell.


Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.
RESETTING PERMS
SETUID ROOT OK
/usr/local/nagiosxi/nom/checkpoints/nagioscore /usr/local/nagiosxi/scripts
tar: Removing leading `/' from member names
/usr/local/nagiosxi/scripts
OUTPUT=/usr/local/nagiosxi/scripts
RETURNCODE=0
.......................
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........................................................





Additionally, in the web gui, if I go to Monitoring Process/Process Info, I see "Monitoring Engine Process.... No data... Last Updated: <foo>". Where normally I should be able to control enabling/disable notifications, etc from there.


Thanks
ej

Can we verify the cmdsubsys cron is running

Code: Select all

ps -ef |grep cmdsubsys

if so, can you tail that log while trying to apply configuration

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[scottwilkerson]

Additionally, in the web gui, if I go to Monitoring Process/Process Info, I see "Monitoring Engine Process.... No data... Last Updated: <foo>". Where normally I should be able to control enabling/disable notifications, etc from there.

Can we run the following from the command line

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start

Then go back and look at this page, if the info doesn't populate we may have a db problem

[emarmonti]

Can we run the following from the command line

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start

Then go back and look at this page, if the info doesn't populate we may have a db problem

After following your instructions, there's still no information on the Process Info page. Is there a a tool built into nagios that validates core database integrity? What else would you suggest?


Thanks

[abrist]

Follow the steps in the document below to verify and repair the mysql database:
http://assets.nagios.com/downloads/nagi ... tabase.pdf

[emarmonti]

I repaired the database per documentation. It finished within 30 seconds. mysqld started up fine with no errors after the repairs.

Unfortunately I'm still experiencing the same behavior. To summarize the issues again:

* The monitor engine process shows "no data" under Process Info
* When scheduling immediate service checks, I get ""Your request was not processed in a timely manner. It may still execute, as the server may be temporarily busy." (though the checks do in fact get scheduled immediately per cmdsubsys.log, XI does not recognize it).
* When applying configuration changes, I get "Waiting for configuration verification....................." and it times out. Though it does reload the config as I realized after looking at cmdsubsys.log, Nagios XI does not recognize it reloaded the config.
* The Quick Find no longer auto-populates the field as I start typing in a hostname.

It seems like Nagios XI is not properly communicating with Nagios core.

This is very frustrating, especially since I followed the documentation when I restored the configuration and it worked perfect before this. Any other ideas?



Thanks

Follow the steps in the document below to verify and repair the mysql database:
http://assets.nagios.com/downloads/nagi ... tabase.pdf

[scottwilkerson]

Is crond running?

Code: Select all

service crond status

Also can you port the output of

Code: Select all

tail -20 /var/log/cron