After upgrade all hosts and service checks stuck in pending

[ks6764]

I have a server that was running 5.2.3 and I now have on 5.4.4.
I saved the previous config files and used the Nagios import tool to import the configs after installing 5.4.4.
The procedure seems to have run fine and I made sure the token is in place as the monitoring of our client servers is passive using nrdp.
Currently, all servers under Hostgroup Overview show as green and ok for everything. If I drill down to a server and its services, I see them, but the Status column has "pending", the "Duration" and "Last Check" columns have "N/A". Lastly, the "Status Information" column has No check results for service yet...
I have restarted all the services (Nagios, MySQL, and Apache) with no issues. I also logged on to one of the client servers being monitored and can run the nrdp based monitoring check as listed under the Nagios crontab and it states that it has sent 15 checks to the server.
What am I missing ??

[tgriep]

If you login to the XI GUI and go to the Admin > Unconfigured Objects menu, do those host and service show up there?
You may need to re-configure them so the XI server will receive the passive checks again.

[ks6764]

I checked the unconfigured objects and there is nothing there to import.

[tgriep]

Can you open the NRDP Server API by opening a Web Browser and use the following URL, replacing xxx.xxx.xxx.xxx for the IP address of the Nagios XI server
SSL not enabled on the Nagios server

Code: Select all

http://xxx.xxx.xxx.xxx/nrdp/

SSL is enabled on the Nagios server

Code: Select all

https://xxx.xxx.xxx.xxx/nrdp/

In the Submit Check Data field, put in the NRDP Token and click on the Submit Check Data button.
Then go to the Admin > Unconfigured Objects menu and do you see the test host called somehost?

[ks6764]

Bad Check results Dir error

[tgriep]

Without knowing if the system is setup to use a ramdisk, I will need you run some commands in the Nagios server and post the output.

Code: Select all

grep checkresults /usr/local/nagios/etc/nagios.cfg
grep check_results_dir /usr/local/nrdp/server/config.inc.php
ls -l /usr/local/nagios/var/spool/
ls -l /var/nagiosramdisk/spool/

[ks6764]

Code: Select all

# grep checkresults /usr/local/nagios/etc/nagios.cfg
check_result_path=/usr/local/nagios/var/spool/checkresults

# grep check_results_dir /usr/local/nrdp/server/config.inc.php
$cfg["check_results_dir"] = "/usr/local/nagios/var/spool/checkresults";

# ls -l /usr/local/nagios/var/spool
total 12
drwsrwsr-- 2 nagios apache 4096 Jun 19 14:12 checkresults
drwxrwxr-x 2 nagios nagios 4096 Jun 19 14:12 perfdata
drwxrwxr-x 2 nagios nagios 4096 Jun 19 14:12 xidpe

/var/nagiosramdisk/spool/ - no such file or directory

[tgriep]

The permissions are not correct for the checkresults folder. Login as root, run the following to fix it.

Code: Select all

chown nagios.nagcmd /usr/local/nagios/var/spool/checkresults
chmod 775 /usr/local/nagios/var/spool/checkresults
chmod g+s /usr/local/nagios/var/spool/checkresults
service nagios restart
service httpd restart

Then login to the NRDP API and see id you can submit a command. Then check and see if the server starts to receive the passive data from your remote hosts.

[ks6764]

I changed the permissions as follows and restarted the services. Still getting the Bad Check Results Dir message.

Code: Select all

# ls -l
total 12
drwxrwsr-x 2 nagios nagcmd 4096 Jun 19 15:17 checkresults
drwxrwxr-x 2 nagios nagios 4096 Jun 19 15:17 perfdata
drwxrwxr-x 2 nagios nagios 4096 Jun 19 15:17 xidpe

As expected, when I run a

Code: Select all

 /usr/local/nrdp/clients/nrds/nrds.pl -H 'myhost'  

from a client box that is being monitored, it returns

Code: Select all

ERROR: The NRDP Server said BAD CHECK RESULTS DIR

[tgriep]

Are any of the folders in the following path a symlink from another folder?

Code: Select all

/usr/local/nagios/var/spool/checkresults

If so edit the /usr/local/nrdp/server/config.inc.php file and update the check_results_dir to point the real location and not the symlink.

Code: Select all

$cfg["check_results_dir"] = "/usr/local/nagios/var/spool/checkresults";

Save and restart the Apache process by running

Code: Select all

service httpd restart

The PHP function the NRDP PHP software uses has issues with symlinks.