Checks pending

[bcdudley]

Last night, we performed maintenance on one of our core switches which knocked out a large section of our network. NagiosXI, as expected reported the outage and lit up with alerts. After service was restored, Nagios refused to perform any updated checks and show that the service was back. We tried to run an immediate check on the blocking host and it would not run. After about 30 minutes of non-responsiveness, we rebooted the Nagios server. After it came back up, every single host and service is showing

"Host check is pending..."
Status Details
Host State: Pending
Duration: N/A
Host Stability: Changing
Current Check: of
Last Check: Never
Next Check: Not scheduled
Host Notes:
Host notifications are disabled
Host notifications are disabled

If I go to the Performance graphs, I get the message "You are not authorized to access this feature. Contact your Nagios XI administrator for more information, or to obtain access to this feature."

Under the advanced tab, I see:

Attribute
State
Action
Active Checks Disabled Enable
Passive Checks Disabled Enable
Notifications Disabled Enable
Flap Detection Disabled Enable
Event Handler Disabled Enable
Performance Data Disabled
Obsession Disabled Enable

And if I try to click on "Re-configure this host", I am sent to the "Configuration Options" screen instead of the host configuration.

I can restore from a previous backup if necessary, but I would hope it would not be. Any ideas?

[slansing]

Hmm, what distro and version of care are you running on? Please also show the output of:

Code: Select all

ll/usr/local/nagios/var/retention.dat

Code: Select all

service nagios status

service crond status

service npcd status

Code: Select all

date

grep "timezone" /etc/php.ini

[bcdudley]

Did it again this morning. I had to reboot the server, then re-apply the configuration to get it going again. Everything is fine now

nagios (pid 5108) is running...
crond (pid 1628) is running...
NPCD running (pid 1638).

Mon Dec 30 14:43:10 CST 2013

; Defines the default timezone used by the date functions
; http://www.php.net/manual/en/datetime.c ... e.timezone
date.timezone = America/Chicago

The Retention.dat file is 25mb and contains potentially sensitive data. The first few lines are:

Code: Select all

info {
created=1388434757
version=3.5.0
last_update_check=1388354907
update_available=1
update_uid=1381717605
last_version=3.5.0
new_version=4.0.2
}
program {
modified_host_attributes=0
modified_service_attributes=0
enable_notifications=1
active_service_checks_enabled=1
passive_service_checks_enabled=1
active_host_checks_enabled=1
passive_host_checks_enabled=1
enable_event_handlers=1
obsess_over_services=0
obsess_over_hosts=0
check_service_freshness=1
check_host_freshness=0
enable_flap_detection=1
enable_failure_prediction=1
process_performance_data=1
global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler
next_comment_id=3420
next_downtime_id=1
next_event_id=130267
next_problem_id=58362
next_notification_id=823686
}

[abrist]

I believe Sam was not looking for the retention.dat content, but for the time stamp / permissions. What is the output of:

Code: Select all

ls -la /usr/local/nagios/var/retention.dat