Allow service notifications when host status is DOWN

[emarton]

Hello,

I'm working on Nagios Core 4.0.8 running on a Centos 7 machine and I noticed that service notifications are not sent in case the host is in a down state.
I know this is logical to have, however in case our setup requires precise service notifications regardless of the host state can this be configured in any way ?
I checked all change logs and documentation of nagios core 4.xx however couldn't find any description of this.
The configuration i'm using doesn't have any dependencies set for either hosts or services and notifications work fine for both host and service as long as the host is not in a down state.

Here are the nagios logs:

Code: Select all

[1458110500] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;SOFT;1;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110520] HOST ALERT: xxx.xxx.com;DOWN;SOFT;1;UNKNOWN - 6 plugins checked, 1 unknown (distributed_check), 5 ok
[1458110560] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;SOFT;2;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110580] HOST ALERT: xxx.xxx.com;DOWN;SOFT;2;UNKNOWN - 6 plugins checked, 1 unknown (distributed_check), 5 ok
[1458110620] SERVICE ALERT: xxx.xxx.com;Check Multi;CRITICAL;HARD;3;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110641] HOST ALERT: xxx.xxx.com;DOWN;HARD;3;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok
[1458110641] HOST NOTIFICATION: contact1;xxx.xxx.com;DOWN;notify-host-by-email;CRITICAL - 6 plugins checked, 1 critical (distributed_check), 5 ok

Nagios debug logs:

Code: Select all

[1458110620.975715] [032.0] [pid=24363] ** Service Notification Attempt ** Host: 'xxx.xxx.com', Service: 'Check Multi', Type: NORMAL, Options: 0, Current State: 2, Last Notification: Wec 31 19:00:00 1969
[1458110620.976508] [032.1] [pid=24363] The host is either down or unreachable, so we won't notify contacts about this service.
[1458110620.976645] [032.0] [pid=24363] Notification viability test failed.  No notification will be sent out.
[1458110641.260207] [032.0] [pid=24363] ** Host Notification Attempt ** Host: 'xxx.xxx.com', Type: NORMAL, Options: 0, Current State: 1, Last Notification: Wed Dec 31 19:00:00 1969
[1458110641.260262] [032.0] [pid=24363] Notification viability test passed.
[1458110641.260275] [032.1] [pid=24363] Current notification number: 1 (incremented)
[1458110641.260288] [032.1] [pid=24363] Host notification will NOT be escalated.
[1458110641.260305] [032.1] [pid=24363] Adding normal contacts for host to notification list.
[1458110641.260536] [032.0] [pid=24363] 1 contacts were notified.  Next possible notification time: Wed Mar 16 03:14:01 2016
[1458110641.260548] [032.0] [pid=24363] 1 contacts were notified.

Any insight to this is greatly appreciated.
Thanks

[rkennedy]

Can you please look at /usr/local/nagios/var/objects.cache file, and post the full host / service definition from it for the one that's causing you this issue?

[emarton]

Hello,

Here are the host and service definitions. Also all other hosts/services have the same behavior probably because all were added the same way.

Code: Select all

define host {
        host_name       xxx.xxx.com
        alias   xxx.xxx.com
        address xx.xx.xx.xx
        check_period    24x7
        check_command   check-host-alive!500,50%!1000,100%!xx.xx.xx.xx,xx.xx.xx.xx!0!0
        contact_groups  customer1group
        notification_period     24x7
        initial_state   o
        importance      0
        check_interval  5.000000
        retry_interval  1.000000
        max_check_attempts      3
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    r,d,u,s
        notifications_enabled   1
        notification_interval   30.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        }

define service {
        host_name       xxx.xxx.com
        service_description     Check Multi
        check_period    24x7
        check_command   check_multi_icmp!20,20%!100,50%!xx.xx.xx.xx,xx.xx.xx.xx!0!0
        contact_groups  customer1group
        notification_period     24x7
        initial_state   o
        importance      0
        check_interval  1.000000
        retry_interval  1.000000
        max_check_attempts      3
        is_volatile     0
        parallelize_check       1
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    r,w,u,c,s
        notifications_enabled   1
        notification_interval   10.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        }

[eloyd]

You can do this a couple of ways, but event handlers come to mind, as they fire every time a service check is completed. Rather than notifying through Nagios directly, you could write a service event handler that always notifies when something is not in an OK state, since service checks are performed when the host is down, they just do not send notifications at that time.

[emarton]

You can do this a couple of ways, but event handlers come to mind, as they fire every time a service check is completed. Rather than notifying through Nagios directly, you could write a service event handler that always notifies when something is not in an OK state, since service checks are performed when the host is down, they just do not send notifications at that time.

I was hoping to do this though nagios as tracking custom scripts etc will complicate things later on, especially on maintenance.
I worked with nagios before a long time ago and then it was sending notifications for the services regardless of the host state, not exactly sure when this changed, I couldn't find anything in the version 4 release notes.
I would assume that if this feature was added some backward compatibility should still exists, but can't find it.

Any other suggestion on this ?
Thanks.

[tmcdonald]

Can you post your nagios.cfg please?

[emarton]

Hi, here it is.
Thanks.

Code: Select all

log_file=/var/log/nagios/nagios.log
object_cache_file=/var/nagiosramdisk/objects.cache
precached_object_file=/var/log/nagios/objects.precache
resource_file=/etc/nagios/private/resource.cfg
status_file=/var/nagiosramdisk/status.dat
status_update_interval=10
nagios_user=nagios
nagios_group=nagios
check_external_commands=1
command_file=/var/spool/nagios/cmd/nagios.cmd
query_socket=/var/spool/nagios/cmd/nagios.qh
lock_file=/var/run/nagios/nagios.pid
temp_file=/var/nagiosramdisk/nagios.tmp
temp_path=/var/nagiosramdisk/tmp
event_broker_options=-1
broker_module=/usr/lib64/mod_gearman2/mod_gearman2.o config=/etc/mod_gearman2/module.conf 
log_rotation_method=d
log_archive_path=/var/log/nagios/archives
use_syslog=1
log_notifications=1
log_service_retries=1
log_host_retries=1
log_event_handlers=1
log_initial_states=0
log_current_states=1
log_external_commands=1
log_passive_checks=0
service_inter_check_delay_method=s
max_service_check_spread=30
service_interleave_factor=s
host_inter_check_delay_method=s
max_host_check_spread=30
max_concurrent_checks=0
check_result_reaper_frequency=10
max_check_result_reaper_time=30
check_result_path=/var/nagiosramdisk/spool/checkresults
max_check_result_file_age=3600
cached_host_check_horizon=15
cached_service_check_horizon=15
enable_predictive_host_dependency_checks=1
enable_predictive_service_dependency_checks=1
soft_state_dependencies=0
auto_reschedule_checks=0
auto_rescheduling_interval=30
auto_rescheduling_window=180
service_check_timeout=60
host_check_timeout=30
event_handler_timeout=30
notification_timeout=30
ocsp_timeout=5
perfdata_timeout=5
retain_state_information=1
state_retention_file=/var/log/nagios/retention.dat
retention_update_interval=60
use_retained_program_state=1
use_retained_scheduling_info=1
retained_host_attribute_mask=0
retained_service_attribute_mask=0
retained_process_host_attribute_mask=0
retained_process_service_attribute_mask=0
retained_contact_host_attribute_mask=0
retained_contact_service_attribute_mask=0
interval_length=60
check_for_updates=1
bare_update_check=0
use_aggressive_host_checking=0
execute_service_checks=1
accept_passive_service_checks=1
execute_host_checks=1
accept_passive_host_checks=1
enable_notifications=1
enable_event_handlers=1
process_performance_data=1
service_perfdata_file=/var/nagiosramdisk/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file

host_perfdata_file=/var/nagiosramdisk/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file

obsess_over_services=0
obsess_over_hosts=0
translate_passive_host_checks=0
passive_host_checks_are_soft=0
check_for_orphaned_services=1
check_for_orphaned_hosts=1
check_service_freshness=1
service_freshness_check_interval=60
service_check_timeout_state=c
check_host_freshness=1
host_freshness_check_interval=60
additional_freshness_latency=15
enable_flap_detection=0
low_service_flap_threshold=5.0
high_service_flap_threshold=20.0
low_host_flap_threshold=5.0
high_host_flap_threshold=20.0
date_format=us
illegal_object_name_chars=`~!$%^&*|'"<>?,()=
illegal_macro_output_chars=`~$&|'"<>
use_regexp_matching=0
use_true_regexp_matching=0
admin_email=nagios@localhost
admin_pager=pagenagios@localhost
daemon_dumps_core=0
use_large_installation_tweaks=1
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
debug_level=32
debug_verbosity=1
debug_file=/var/log/nagios/nagios.debug
max_debug_file_size=1000000
allow_empty_hostgroup_assignment=0
cfg_dir=/etc/nagios/objects/local

[tgriep]

Take a look at this earlier post to see if there is anything that could help.
https://support.nagios.com/forum/viewto ... 20&t=27796

[tmcdonald]

In addition to what @tgriep said, I dug into the notification code and found this:

https://github.com/NagiosEnterprises/na ... ons.c#L567

I could have sworn as well that service notifications could go out when the host is down, but we must have been collectively mistaken. This is something I might consider adding in a flag for.

I am going to check in with the devs on this and get confirmation that this is in fact expected behavior.

[eloyd]

Funny, I knew that. That's why I said event handlers.