Service check is pending...Forever

[toodaly]

I am using Nagios XI version 2012R2.9

Here are the tester's notes:
Using Nagios XI to reconfigure a service to demonstrate effects on service status and notifications:
1. At the Service Status Detail screen for a service (e.g., Memory Usage) on a host (e.g. Los Angeles), select the Configure tab, then select "Re-configure this service".
2. At the Configure Service Memory Usage Los Angeles screen, enter 1 in each of the three entry fields, and change the Monitor Service command (e.g., check_nrpe!check_mem!-a '-w 99 -c 98'). Click the Update button.
3. The Configure Service screen displays "Configuration submitted for processing..."; "Waiting for configuration verification...", "Configuration applied successfully."
4. In the Configuration window, press the "Continue" button.
5. The Service Status Detail window - Overview tab is displayed with a status of "Service check is pending...", with Status Details: Service State: Pending; ... Last Check: Never; Next Check: Not Scheduled.

This configuration has ~2000 hosts, ~7000 services from 40 remote sites. Understandably things move slow. The work-around we've been using is once the "Configuration applied successfully" message is displayed, wait about 30 seconds before pressing the "Continue" button. Everything seems fine after that. I assume this is the time it takes for Nagios XI to do it's reconfiguration and restart of the Nagios services.

The tester feels this is a potential bug and Nagios XI should at least verify that all of its services have restarted correctly be it displays "Configuration applied successfully."

What are your thoughts on this?

Thanks.

[jdalrymple]

That sounds like a bug, but it also sounds vaguely (albeit not exactly) like a known phenomena.

In moderate to large size environments the apply config process will put the database in a bit of a lag state and as a result the XI interface doesn't have it's most current data. This is usually easy to spot by looking at the system status and seeing that the last 3 icons are all the blue i. Once the db catches up all those icons become green checks again.

The solution is to either make your database or database connection function faster.

Back to your problem - this does sound like a bug but it seems weird that we've never seen it before.

If you recreate the issue can you check /usr/local/nagios/var/objects.cache and verify that the service in question is getting a valid check_command?

[tmcdonald]

I actually don't think this is a bug at all. The configuration did successfully apply, meaning the files were written to disk and the nagios service was restarted. It's not saying "Configuration successfully applied and service is back up to full speed". Once the message appears, the configs have been written and the service restarted, but that does not mean that all of the checks you have added have had a chance to run.

Also, this is the 2012 version. In 2014 we made some big improvements to the Apply Config speed, and in XI 5 we sped up the interface in other places. There's not much we can do for a bug report against a software version from 3 years ago. But even in the current version, it will take some time for checks to be run initially.

I might be misunderstanding the issue, but this does not sound like a bug. If you can provide some screenshots of this in XI 5 we can possibly take some action.

[toodaly]

I was only allowed to attach one file and it had to be less than 1MB. I put what screen shots I could into a .pdf with captions and zipped it (see attached).

Let me know if you have any questions with it.

Thanks.

[jdalrymple]

I can't recreate the issue, but my system also doesn't have the load yours does.

Is it a true statement that after the change you can wait an eternity and the interface never stabilizes? Namely that bit where active checks, passive checks, notifications all read. I mean I assume those aren't supposed to all be red?

And again, check objects.cache when the service is in a problem state.

[toodaly]

True, it will stay like that as long as I am on that Service Status Detail web page, going back and forth through the tabs (Overview, Performance Graphs, Advanced, and Configure). However, once I leave (i.e. click Service Detail -> <recently changed service>) and go back to the same Service Status Detail page, all is fine and checks are green.

Sorry, I forgot to add the objects.cache information. I captured the file before the change, after the change without waiting for the System Ok to go back to all green, and again after making changes with waiting for the System Ok to go back to all green. I did a diff between all three files, the only things that changed were the Created <Date/Time> line and the threshold (I changed the CPU warning threshold) line.

I'm seeing the issue in my test environment where I have ~2000 host and ~7000 services configured, but only ~2% of the hardware is connected (~40 physical hosts and associated services) in the green. The rest show up as red. It might be a database size issue that you can probably recreate with enough dummy hosts and services.

Bottom line is the tester who noted the issue, would like a bug tracking number or what version of Nagios XI this was fixed under (for potential upgrade down the road) to put in his report.

Thanks.

[jdalrymple]

Bottom line is the tester who noted the issue, would like a bug tracking number or what version of Nagios XI this was fixed under (for potential upgrade down the road) to put in his report.

I don't see a reference to the bug in our internal tracker, as such I can't offer the information you're seeking here. I'd be happy to put one in, however we'd have to know that the bug still exists in current versions. We don't do patch releases for older versions, or at least it's not something we've done yet.

[toodaly]

I was able to get Nagios XI 5 installed in my test environment with the same configuration. Unfortunately, I had the same issue (see attachment). I don't have a login for the bug tracker, so I'll take jdalrymple up on his offer to put one in.

My suggestion for a potential fix would be to bring back what was in Nagios XI 2012R2.9 (3 green checks)
1) Configuration submitted for processing
2) Waiting for configuration verification
3) Configuration applied successfully
and add a fourth:
4) Waiting for backend to restart
before presenting the user with the Continue button.

Thanks.

[jdalrymple]

I'm still unable to recreate.

Have you customized your nagios.cfg? I feel like there is something larger at play here. Changing a service in place should never clear out old status - not that I know of anyway.

Regarding the 3 checks - they're still there, they've just been combined into 1 check. If any single part of the 3 check process fails it will alert you.

[toodaly]

The configuration files for the telephone system were created by an engineer who is used to working with Nagios Core. His installer loads those devices directly into the Nagios database. I don't see them in Nagios XI (MySQL), but they show up in the host, hostgroup, service, servicegroup views of Nagios XI. The only difference between the nagios.cfg and a base install version of nagios.cfg is:
#{ nagios.configuration
cfg_dir=/usr/local/nagios/etc/_
#} nagios.configuration
that is appended to the end of nagios.cfg. etc/_ is where he keeps his host and service configuration files for the telephone devices.
Also, using the XI import prep tool gives errors when running on his .cfg files.