Page 1 of 2
Allow service notifications when host status is DOWN
Posted: Wed Mar 16, 2016 4:43 am
by emarton
Hello,
I'm working on Nagios Core 4.0.8 running on a Centos 7 machine and I noticed that service notifications are not sent in case the host is in a down state.
I know this is logical to have, however in case our setup requires precise service notifications regardless of the host state can this be configured in any way ?
I checked all change logs and documentation of nagios core 4.xx however couldn't find any description of this.
The configuration i'm using doesn't have any dependencies set for either hosts or services and notifications work fine for both host and service as long as the host is not in a down state.
Here are the nagios logs:
Code: Select all
[1458110500] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;SOFT;1;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110520] HOST ALERT: xxx.xxx.com;DOWN;SOFT;1;UNKNOWN - 6 plugins checked, 1 unknown (distributed_check), 5 ok
[1458110560] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;SOFT;2;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110580] HOST ALERT: xxx.xxx.com;DOWN;SOFT;2;UNKNOWN - 6 plugins checked, 1 unknown (distributed_check), 5 ok
[1458110620] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;HARD;3;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110641] HOST ALERT: xxx.xxx.com;DOWN;HARD;3;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110641] HOST NOTIFICATION: contact1;xxx.xxx.com;DOWN;notify-host-by-email;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
Nagios debug logs:
Code: Select all
[1458110620.975715] [032.0] [pid=24363] ** Service Notification Attempt ** Host: 'xxx.xxx.com', Service: 'Check Multi', Type: NORMAL, Options: 0, Current State: 2, Last Notification: Wec 31 19:00:00 1969
[1458110620.976508] [032.1] [pid=24363] The host is either down or unreachable, so we won't notify contacts about this service.
[1458110620.976645] [032.0] [pid=24363] Notification viability test failed. No notification will be sent out.
[1458110641.260207] [032.0] [pid=24363] ** Host Notification Attempt ** Host: 'xxx.xxx.com', Type: NORMAL, Options: 0, Current State: 1, Last Notification: Wed Dec 31 19:00:00 1969
[1458110641.260262] [032.0] [pid=24363] Notification viability test passed.
[1458110641.260275] [032.1] [pid=24363] Current notification number: 1 (incremented)
[1458110641.260288] [032.1] [pid=24363] Host notification will NOT be escalated.
[1458110641.260305] [032.1] [pid=24363] Adding normal contacts for host to notification list.
[1458110641.260536] [032.0] [pid=24363] 1 contacts were notified. Next possible notification time: Wed Mar 16 03:14:01 2016
[1458110641.260548] [032.0] [pid=24363] 1 contacts were notified.
Any insight to this is greatly appreciated.
Thanks
Re: Allow service notifications when host status is DOWN
Posted: Wed Mar 16, 2016 11:41 am
by rkennedy
Can you please look at /usr/local/nagios/var/objects.cache file, and post the full host / service definition from it for the one that's causing you this issue?
Re: Allow service notifications when host status is DOWN
Posted: Thu Mar 17, 2016 2:11 am
by emarton
Hello,
Here are the host and service definitions. Also all other hosts/services have the same behavior probably because all were added the same way.
Code: Select all
define host {
host_name xxx.xxx.com
alias xxx.xxx.com
address xx.xx.xx.xx
check_period 24x7
check_command check-host-alive!500,50%!1000,100%!xx.xx.xx.xx,xx.xx.xx.xx!0!0
contact_groups customer1group
notification_period 24x7
initial_state o
importance 0
check_interval 5.000000
retry_interval 1.000000
max_check_attempts 3
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options r,d,u,s
notifications_enabled 1
notification_interval 30.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
}
define service {
host_name xxx.xxx.com
service_description Check Multi
check_period 24x7
check_command check_multi_icmp!20,20%!100,50%!xx.xx.xx.xx,xx.xx.xx.xx!0!0
contact_groups customer1group
notification_period 24x7
initial_state o
importance 0
check_interval 1.000000
retry_interval 1.000000
max_check_attempts 3
is_volatile 0
parallelize_check 1
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options r,w,u,c,s
notifications_enabled 1
notification_interval 10.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
}
Re: Allow service notifications when host status is DOWN
Posted: Thu Mar 17, 2016 10:34 am
by eloyd
You can do this a couple of ways, but event handlers come to mind, as they fire every time a service check is completed. Rather than notifying through Nagios directly, you could write a service event handler that always notifies when something is not in an OK state, since service checks are performed when the host is down, they just do not send notifications at that time.
Re: Allow service notifications when host status is DOWN
Posted: Thu Mar 17, 2016 10:56 am
by emarton
eloyd wrote:You can do this a couple of ways, but event handlers come to mind, as they fire every time a service check is completed. Rather than notifying through Nagios directly, you could write a service event handler that always notifies when something is not in an OK state, since service checks are performed when the host is down, they just do not send notifications at that time.
I was hoping to do this though nagios as tracking custom scripts etc will complicate things later on, especially on maintenance.
I worked with nagios before a long time ago and then it was sending notifications for the services regardless of the host state, not exactly sure when this changed, I couldn't find anything in the version 4 release notes.
I would assume that if this feature was added some backward compatibility should still exists, but can't find it.
Any other suggestion on this ?
Thanks.
Re: Allow service notifications when host status is DOWN
Posted: Thu Mar 17, 2016 4:30 pm
by tmcdonald
Can you post your nagios.cfg please?
Re: Allow service notifications when host status is DOWN
Posted: Fri Mar 18, 2016 12:14 am
by emarton
Hi, here it is.
Thanks.
Code: Select all
log_file=/var/log/nagios/nagios.log
object_cache_file=/var/nagiosramdisk/objects.cache
precached_object_file=/var/log/nagios/objects.precache
resource_file=/etc/nagios/private/resource.cfg
status_file=/var/nagiosramdisk/status.dat
status_update_interval=10
nagios_user=nagios
nagios_group=nagios
check_external_commands=1
command_file=/var/spool/nagios/cmd/nagios.cmd
query_socket=/var/spool/nagios/cmd/nagios.qh
lock_file=/var/run/nagios/nagios.pid
temp_file=/var/nagiosramdisk/nagios.tmp
temp_path=/var/nagiosramdisk/tmp
event_broker_options=-1
broker_module=/usr/lib64/mod_gearman2/mod_gearman2.o config=/etc/mod_gearman2/module.conf
log_rotation_method=d
log_archive_path=/var/log/nagios/archives
use_syslog=1
log_notifications=1
log_service_retries=1
log_host_retries=1
log_event_handlers=1
log_initial_states=0
log_current_states=1
log_external_commands=1
log_passive_checks=0
service_inter_check_delay_method=s
max_service_check_spread=30
service_interleave_factor=s
host_inter_check_delay_method=s
max_host_check_spread=30
max_concurrent_checks=0
check_result_reaper_frequency=10
max_check_result_reaper_time=30
check_result_path=/var/nagiosramdisk/spool/checkresults
max_check_result_file_age=3600
cached_host_check_horizon=15
cached_service_check_horizon=15
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
soft_state_dependencies=0
auto_reschedule_checks=0
auto_rescheduling_interval=30
auto_rescheduling_window=180
service_check_timeout=60
host_check_timeout=30
event_handler_timeout=30
notification_timeout=30
ocsp_timeout=5
perfdata_timeout=5
retain_state_information=1
state_retention_file=/var/log/nagios/retention.dat
retention_update_interval=60
use_retained_program_state=1
use_retained_scheduling_info=1
retained_host_attribute_mask=0
retained_service_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
interval_length=60
check_for_updates=1
bare_update_check=0
use_aggressive_host_checking=0
execute_service_checks=1
accept_passive_service_checks=1
execute_host_checks=1
accept_passive_host_checks=1
enable_notifications=1
enable_event_handlers=1
process_performance_data=1
service_perfdata_file=/var/nagiosramdisk/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file
host_perfdata_file=/var/nagiosramdisk/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file
obsess_over_services=0
obsess_over_hosts=0
translate_passive_host_checks=0
passive_host_checks_are_soft=0
check_for_orphaned_services=1
check_for_orphaned_hosts=1
check_service_freshness=1
service_freshness_check_interval=60
service_check_timeout_state=c
check_host_freshness=1
host_freshness_check_interval=60
additional_freshness_latency=15
enable_flap_detection=0
low_service_flap_threshold=5.0
high_service_flap_threshold=20.0
low_host_flap_threshold=5.0
high_host_flap_threshold=20.0
date_format=us
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
illegal_macro_output_chars=`~$&|'"<>
use_regexp_matching=0
use_true_regexp_matching=0
admin_email=nagios@localhost
admin_pager=pagenagios@localhost
daemon_dumps_core=0
use_large_installation_tweaks=1
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
debug_level=32
debug_verbosity=1
debug_file=/var/log/nagios/nagios.debug
max_debug_file_size=1000000
allow_empty_hostgroup_assignment=0
cfg_dir=/etc/nagios/objects/local
Re: Allow service notifications when host status is DOWN
Posted: Fri Mar 18, 2016 3:09 pm
by tgriep
Take a look at this earlier post to see if there is anything that could help.
https://support.nagios.com/forum/viewto ... 20&t=27796
Re: Allow service notifications when host status is DOWN
Posted: Fri Mar 18, 2016 3:17 pm
by tmcdonald
In addition to what
@tgriep said, I dug into the notification code and found this:
https://github.com/NagiosEnterprises/na ... ons.c#L567
I could have sworn as well that service notifications could go out when the host is down, but we must have been collectively mistaken. This is something I might consider adding in a flag for.
I am going to check in with the devs on this and get confirmation that this is in fact expected behavior.
Re: Allow service notifications when host status is DOWN
Posted: Fri Mar 18, 2016 4:00 pm
by eloyd
Funny, I knew that. That's why I said event handlers.
