Allow service notifications when host status is DOWN

[emarton]

In addition to what @tgriep said, I dug into the notification code and found this:

https://github.com/NagiosEnterprises/na ... ons.c#L567

I could have sworn as well that service notifications could go out when the host is down, but we must have been collectively mistaken. This is something I might consider adding in a flag for.

I am going to check in with the devs on this and get confirmation that this is in fact expected behavior.

Thank you guys for the insight in this, i knew for sure at some older version the host status didn't matter when sending service notification, which was indeed annoying and generated a lot of emails in case the host was down.
In the setup i'm doing now, I'm not that much interested in the host status, but i need to get notifications for each service under the host regardless of the host state.
Adding a flag or some optional setting though which this behavior can be controlled would be excellent.
Please update me back once you hear from the devs, if there will be no change to the current behavior then I would need to go with event handlers to get the desired behavior.

Thanks.

[lmiltchev]

In the setup i'm doing now, I'm not that much interested in the host status, but i need to get notifications for each service under the host regardless of the host state.

If you don't care about the host's state, you could disable active checks for this host. Make sure the host is in "UP" state. If not - you can always submit a passive check result ("UP") to change the state.
Alternatively, you could replace the existing host check command with "check_dummy", and pass "0" as an argument.
In both cases, the host's status will always be "UP", and service notifications will be sent.

Adding a flag or some optional setting though which this behavior can be controlled would be excellent.
Please update me back once you hear from the devs...

This will take some time to get implemented (if ever). You are welcome to post a feature request on our tracker here:
http://tracker.nagios.org

[eloyd]

Bingo. That's what I'd do if you don't want to write event handlers. Use the check_dummy command to always have the host come back as OK. You can use a check_ping service check to actually tell you if the host is not reachable (since that's also the default host check).

Then you'll get service notifications every time, regardless of the state of the host.

If you want to get notified for EVERY service check result when it changes state (not just after Nagios decides it's been bad long enough to warrant a notification) you'll want to look at the is_volatile service definition parameter as well.

[bwallace]

Looks like lmiltchev and eloyd have come up with a solution - were there any other questions or may we lock this thread?

[emarton]

Just one last question,
I was testing the event handlers in an attempt to replicate the default nagios notification.
So I added the following in the configuration:

Command:

Code: Select all

define command {
        command_name    event-notify
        command_line    $USER1$/eventhandlers/event-notify $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$ $NOTIFICATIONTYPE$ $SERVICEDESC$ $HOSTALIAS$ $HOSTADDRESS$ $LONGDATETIME$ $SERVICEOUTPUT$ $NOTIFICATIONTYPE$ $CONTACTEMAIL$
        }

Service:

Code: Select all

define service {
        host_name       xxx.xxx.com
        service_description     Check Multi Event
        check_period    24x7
        check_command   check_multi_icmp!20,20%!100,50%!xx.xx.xx.xx,xx.xx.xx.xx!0!0
        event_handler   event-notify
        contact_groups  customer1group
        notification_period     24x7
        initial_state   o
        importance      0
        check_interval  1.000000
        retry_interval  1.000000
        max_check_attempts      3
        is_volatile     0
        parallelize_check       1
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    r,w,u,c,s
        notifications_enabled   1
        notification_interval   10.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        }

I modified the event handler example for the doc to this:

Code: Select all

#!/bin/sh
# What state is the HTTP service in?
case "$1" in
OK)
	;;
WARNING)
	;;
UNKNOWN)
	;;
CRITICAL)
	case "$2" in

	SOFT)
		;;
				
	HARD)
		# Notify contacts, service in Hard Critical state, using notify-by-email command from nagios
		/usr/bin/printf "%b" "***** Nagios *****\\n\\nNotification Type: $4\\n\\nService: $5\\nHost: $6\\nAddress: $7\\nState: $1\\n\\nDate/Time: $8\\n\\nAdditional Info:\\n\\n$9" | /bin/mail -s "** $10 Service Alert: $6/$5 is $1 **" $11
		;;
	esac
	;;
esac
exit 0

The script is executed correctly, when the service state goes to critical / hard, however the variables are not passed over (except the first 3). I know the CONTACTEMAIL, NOTIFICATIONTYPE and some other macros are not available for service event handlers, but is there any way to get the same service notification as nagios has using event handlers ?
I do have this in the main nagios configuration, not sure if that's the cause though: enable_environment_macros=0

[emarton]

Actually I found the answer to the problem, you cannot use contact macros for service event handlers.
The only way I found to pass over contacts is by using the $CONTACTGROUPMEMBERS:customer1group$ on-demand macro, however for this the contact group name needs to be hardcoded in the event handler command, so this is not a solution for me as I would need to create individual commands for each service event handler.
From what I found you cannot pass $ARGn$ to event_handler definition inside the service definition, any ideas if this can be done through some other method ?

[eloyd]

You can use the nagios API to query the list of contacts for a particular service if you're using 4.0.7 or higher. See https://labs.nagios.com/2014/06/19/expl ... -7-part-1/ for details. You're going to hate me for this, but once you master this sort of thing below, then all the configuration information within Nagios is available to you at the command line as well.

Here is an example to grab the JSON data that shows the contacts for a service:

Code: Select all

http://NAGIOSSERVER/nagios/cgi-bin/objectjson.cgi?query=service&hostname=SOMEHOST&servicedescription=SOMESERVICE

This will return JSON data that looks like this the big list I've posted at the bottom. You could then parse this in your event handler to extract the line after "contacts" (in bash, that would be: contacts=`echo $JSON_OUTPUT | egrep -A 1 "\"contacts\""`), and then use the same APIs to go get each contact's email address with:

Code: Select all

http://NAGIOSSERVER/nagios/cgi-bin/objectjson.cgi?query=contact&contactname=SOMECONTACT

Then grab the "email" field, which would be an email addres and use it in your event handler to send email.

Code: Select all

http://NAGIOSSERVER/nagios/cgi-bin/objectjson.cgi?query=service&hostname=SOMEHOST&servicedescription=SOMESERVICE

{
    "format_version": 0,
    "result": {
        "query_time": 1458734327000,
        "cgi": "objectjson.cgi",
        "user": "nagiosadmin",
        "query": "service",
        "query_status": "released",
        "program_start": 1458643403000,
        "last_data_update": 1458643403000,
        "type_code": 0,
        "type_text": "Success",
        "message": ""
    },
    "data": {
        "service": {
            "host_name": "SOMEHOST",
            "description": "SOMESERVICE",
            "display_name": "SOMESERVICE",
            "check_command": "check_nrpe!check_init_service!httpd",
            "event_handler": "",
            "initial_state": 0,
            "check_interval": 5,
            "retry_interval": 1,
            "max_attempts": 4,
            "parallelize": true,
            "contact_groups": [
                "voip-support"
            ],
            "contacts": [
                "SOMECONTACT"
            ],
            "notification_interval": 60,
            "first_notification_delay": 0,
            "notifications_options": 15,
            "stalking_options": 0,
            "is_volatile": false,
            "notification_period": "24x7",
            "check_period": "24x7",
            "flap_detection_enabled": true,
            "low_flap_threshold": 0,
            "high_flap_threshold": 0,
            "flap_detection_options": -1,
            "process_performance_data": true,
            "check_freshness": false,
            "freshness_threshold": 0,
            "accept_passive_checks": true,
            "event_handler_enabled": true,
            "checks_enabled": true,
            "retain_status_information": true,
            "retain_nonstatus_information": true,
            "notifications_enabled": true,
            "obsess": true,
            "hourly_value": 0,
            "parents": [

            ],
            "children": [

            ],
            "notes": "",
            "notes_url": "",
            "action_url": "",
            "icon_image": "",
            "icon_image_alt": "",
            "custom_variables": {

            }
        }
    }
}

[emarton]

Thanks Eric,

I never though about the API
We do have nagrestconf installed and we are going to use that to setup services. hosts etc
And yes, it can also be used to get the configuration values in json just like you mentioned with the nagios API.
Thank you all for the help on this, thread can be locked.

[eloyd]

Excellent! Glad I could help. Feel free to nominate me for an MVP award like it says in my signature. See you at the Nagios World Conference in September!

[rkennedy]

Thanks @eloyd!

Closing this one out, but feel free to make a new thread if you ever need any assistance in the future!