Scheduled ping checks not executing

[d.ricks]

We have multiple Nagios servers set up that all execute all service checks properly except our ping checks which are constantly stuck in a pending state. A forced check works, but then subsequent scheduled checks still never execute. I am including the config files associated with these checks to provide information on how they are set up:
define service {
host_name HOSTNAME
service_description Ping
servicegroups Services
use xiwizard_linuxserver_ping_service
max_check_attempts 1
check_interval 5
retry_interval 1
check_period 24x7
notification_interval 15
notification_period 24x7
notifications_enabled 0
contact_groups siteops
notification_options w,c
_xiwizard nrpe
register 1
}

From servicetemplates.cfg:

define service {
name xiwizard_mailserver_ping_service
use xiwizard_generic_service
check_command check_xi_service_ping!3000.0!80%!5000.0!100%
register 0

}

define service {
name xiwizard_generic_service
check_command check_xi_service_none!!!!!!!!
is_volatile 0
max_check_attempts 5
check_interval 5
retry_interval 1
active_checks_enabled 1
passive_checks_enabled 1
check_period xi_timeperiod_24x7
parallelize_check 1
obsess_over_service 1
check_freshness 0
event_handler_enabled 1
flap_detection_enabled 1
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
notification_interval 60
notification_period xi_timeperiod_24x7
notifications_enabled 1
register 0

}


From commands.cfg:

define command {
command_name check_xi_service_ping
command_line $USER1$/check_icmp -H $HOSTADDRESS$ -w $ARG1$,$ARG2$ -c $ARG3$,$ARG4$ -p 5
}

define command {
command_name check_icmp
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$
}

define command {
command_name check_ping
command_line $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}

from timeperiods.cfg:

define timeperiod {
timeperiod_name xi_timeperiod_24x7
alias 24x7
name xi_timeperiod_24x7
sunday 00:00-24:00
monday 00:00-24:00
tuesday 00:00-24:00
wednesday 00:00-24:00
thursday 00:00-24:00
friday 00:00-24:00
saturday 00:00-24:00
}

This is the only check on our servers that is not functioning properly, any help would be greatly appreciated

[rkennedy]

Can you navigate to /usr/local/nagios/var/ and open up objects.cache? Then, look for the service in question, and post the full definition. This will show us what's currently running and inherited through the template.

[d.ricks]

From objects.cache:

define service {
host_name HOSTNAME
service_description Ping
check_period 24x7
check_command check-host-alive!3000.0!80%!5000.0!100%!!!!
contact_groups CONTACT GROUP
notification_period 24x7
initial_state o
importance 0
check_interval 5.000000
retry_interval 1.000000
max_check_attempts 1
is_volatile 0
parallelize_check 1
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options w,c
notifications_enabled 0
notification_interval 15.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
_XIWIZARD nrpe
}

[bwallace]

That objects.cache looks good, but unfortunately it does not shed any light on the issue.
Can you post status.dat and nagios.log?
These should tell us the last time these checks were run and well, the status, among other things. Both files are found under:

/usr/local/nagios/var/

At a glance, in the UI, Home > Service Detail, what does the last check time say for these checks? By any chance does the last check time coincide with the actual day they were added to XI?

[lmiltchev]

Can you show us the definitions of the "xiwizard_linuxserver_ping_service" template, and the "check-host-alive" command?

[d.ricks]

This is from the status.dat file for one of the hosts in pending status:

servicestatus {
host_name=HOSTNAME
service_description=Ping
modified_attributes=0
check_command=check_xi_host_ping!3000.0!80%!5000.0!100%!!!!
check_period=24x7
notification_period=24x7
check_interval=5.000000
retry_interval=1.000000
event_handler=
has_been_checked=0
should_be_scheduled=1
check_execution_time=0.000
check_latency=0.000
check_type=0
current_state=0
last_hard_state=0
last_event_id=0
current_event_id=0
current_problem_id=0
last_problem_id=0
current_attempt=1
max_attempts=1
state_type=1
last_state_change=0
last_hard_state_change=0
last_time_ok=0
last_time_warning=0
last_time_unknown=0
last_time_critical=0
plugin_output=
long_plugin_output=
performance_data=
last_check=0
next_check=1460742843
check_options=0
current_notification_number=0
current_notification_id=0
last_notification=0
next_notification=0
no_more_notifications=0
notifications_enabled=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=1
process_performance_data=1
obsess=1
last_update=1460742823
is_flapping=0
percent_state_change=0.00
scheduled_downtime_depth=0
_XIWIZARD=0;nrpe
}

and Nagios.log:

[1460692800] CURRENT SERVICE STATE: HOSTNAME;Ping;OK;HARD;1;

From the UI the last check time for these services are either the date and time when the service was manually forced or is listing as N/A

[d.ricks]

Sorry, I posted the wrong template definition the first time, here is xiwizard_linuxserver_ping_service:

define service {
name xiwizard_linuxserver_ping_service
use xiwizard_generic_service
check_command check_xi_host_ping!3000.0!80%!5000.0!100%!!!!
event_handler_enabled 1
register 0

}

[ssax]

Please post a screenshot of your Admin > Monitoring Engine Status page.

Thank you

[d.ricks]

Admin>Monitoring Engine Status:

MonitoringEngine.png

[lmiltchev]

Did you disable active checks on this service by accident? Go to Home->Service Detail, click on the "problem" Ping service, go to "Advanced" tab, and show us a screenshot of this page.

Is opening a new support ticket in our email ticketing system an option for you? We may need to schedule a remote session to further troubleshoot this issue. Thank you!