Separate notification email for specific service

[bwatsonfc]

I'm trying to send a specific email for a specific service rather then the default/global service notification template. I have configured the following check for Uptime

Code: Select all

check_xi_service_snmp! -o 'sysUpTime.0' -C 'community' -P 2c -l Uptime -w 0:4294607296 -c 25920000:0

The goal here being, when it enters the warning state just before an SNMP rollover occurs, it sends an email that states something along the lines

"Within ~1 hour an SNMP rollover will occur setting this counter to 0 and making it appear as if the device rebooted"

And when uptime is less than 3 days and the service enters a critical state have the email advise

"Uptime is less than 3 days, if there was no preceding email for this device warning of an SNMP rollover please login and check the device"

However I cannot figure out how to create a custom warning/critical alert for my "Uptime" service or what the best way would be to apply that to all of the uptime services on different devices.

Nagios XI Enterprise 5.8.3

Thank you for any help that can be provided.

[gsmith]

Hi

You can use a custom event handler to fire off a script that that sends different emails to
different recipients based on the $HOSTSTATE.
See https://assets.nagios.com/downloads/nag ... ios-XI.pdf
to get started.
I'll keep this post open in case you have questions on the above doc.

Thanks

[bwatsonfc]

I've been trying to follow the instructions from the above link. I have created the command "uptime-event-handler" with command line

Code: Select all

$USER1$/uptime-event-handler.sh $SERVICESTATE$ $SERVICESTATETYPE$ $NOTIFICATIONTYPE$ $SERVICEDESC$ $HOSTALIAS$ $HOSTADDRESS$ $LONGDATETIME$ $SERVICEOUTPUT$ $CONTACTEMAIL$

I have set the event handler drop down to "uptime-event-handler" and selected the "On" box for event handler enabled.

The contents of uptime-event-handler.sh are as follows

Code: Select all

#!/bin/bash
SERVICESTATE=$1
SERVICESTATETYPE=$2
NOTIFICATIONTYPE=$3
SERVICEDESC=$4
HOSTALIAS=$5
HOSTADDRESS=$6
LONGDATETIME=$7
SERVICEOUTPUT=$8
CONTACTEMAIL=$9

# The message below is to warn of SNMP rollover events. Warning threshold on Uptime check is one hour before rollover occurs.

if [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "WARNING" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nUptime service has entered a Warning state indicating that an SNMP rollover event will occur in the next ~60 minutes. This will cause the SNMP uptime counter to reset to 0 making it *appear* as the device rebooted." | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

elif [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "CRITICAL" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nUptime has entered a critical state indicating the device has been up for less than three days. If there was no Warning email for a rollover event on this device in the last ~60 minutes, then please check the device." | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

elif [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "OK" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

fi

However when I force the service into a HARD WARNING state I only get the default Service warning email. Not the message specified above.

[ssax]

Do you see the failures in /var/log/maillog?

What is the output of this command?

Code: Select all

alternatives --display mta

[bwatsonfc]

I see no failures in /var/log/maillog only the success message for the standard/generic service warning email so I assume the script isn't processing correctly to call sendmail?

Code: Select all

mta - status is auto.
 link currently points to /usr/sbin/sendmail.postfix
/usr/sbin/sendmail.postfix - priority 30
 slave mta-pam: /etc/pam.d/smtp.postfix
 slave mta-mailq: /usr/bin/mailq.postfix
 slave mta-newaliases: /usr/bin/newaliases.postfix
 slave mta-rmail: /usr/bin/rmail.postfix
 slave mta-sendmail: /usr/lib/sendmail.postfix
 slave mta-mailqman: /usr/share/man/man1/mailq.postfix.1.gz
 slave mta-newaliasesman: /usr/share/man/man1/newaliases.postfix.1.gz
 slave mta-aliasesman: /usr/share/man/man5/aliases.postfix.5.gz
 slave mta-sendmailman: /usr/share/man/man1/sendmail.postfix.1.gz
Current `best' version is /usr/sbin/sendmail.postfix.

Do I need to somehow call xi-notification-service-handler for the $CONTACTEMAIL$ to process correctly?

[ssax]

It's likely not working because the $CONTACTEMAIL$ is not available in the event handler context:

https://assets.nagios.com/downloads/nag ... olist.html

To get around that you would need to hard code the email address in there, I would also surround some of them with single-quotes to prevent bash interpreter issues:

Code: Select all

$USER1$/uptime-event-handler.sh $SERVICESTATE$ $SERVICESTATETYPE$ $NOTIFICATIONTYPE$ '$SERVICEDESC$' '$HOSTALIAS$' $HOSTADDRESS$ '$LONGDATETIME$' '$SERVICEOUTPUT$' [email protected]

What is the output of this command against the script?

Code: Select all

ls -l /usr/local/nagios/libexec/uptime-event-handler.sh

The mail command in the provided script will use the backend MTA to send the notification which happens to be postfix in this case, in order for that to work you would need to setup postfix to relay through your SMTP server (if you have one). Do you have postfix already setup to relay?

Please send me a copy of your profile, you can download it from Admin > System Profile by clicking the Download Profile button.

[bwatsonfc]

As soon as I hard-coded the email in the script then it worked, so I suspect the script is functioning as expected. Some of the information came through oddly, but I think that may be due to my numbering on the variables. The email however came from a different address then my normal alert emails. I assume that's because /usr/bin/mail is going through postfix and is using that configuration as opposed to the email configuration in XI.

Is there a way I can have the script call the "xi_service_notification_handler" command, or a copy of that command made specifically for this email/script, so that the email address does not have to be hard coded and so it gets sent using the XI email configuration?

[ssax]

You should be able to add -r [email protected] to this portion of the event handler mail commands:

Code: Select all

/bin/mail -s

So:

Code: Select all

#!/bin/bash
SERVICESTATE=$1
SERVICESTATETYPE=$2
NOTIFICATIONTYPE=$3
SERVICEDESC=$4
HOSTALIAS=$5
HOSTADDRESS=$6
LONGDATETIME=$7
SERVICEOUTPUT=$8
CONTACTEMAIL=$9

# The message below is to warn of SNMP rollover events. Warning threshold on Uptime check is one hour before rollover occurs.

if [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "WARNING" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nUptime service has entered a Warning state indicating that an SNMP rollover event will occur in the next ~60 minutes. This will cause the SNMP uptime counter to reset to 0 making it *appear* as the device rebooted." | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

elif [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "CRITICAL" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n\nUptime has entered a critical state indicating the device has been up for less than three days. If there was no Warning email for a rollover event on this device in the last ~60 minutes, then please check the device." | /bin/mail -r [email protected] -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

elif [[ "$SERVICESTATETYPE" == "HARD"  && "$SERVICESTATE" == "OK" ]]; then /usr/bin/printf "%b" "***** Nagios Monitor XI Alert *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$" | /bin/mail -r [email protected] -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$

fi

[bwatsonfc]

Thank you. I was able to use the -r option to specify the from address and modified the body of the email to match the default XI notification other than the added statements regarding uptime. I had to modify my variables as well, it appears that $NOTIFICATIONTYPE$ is not passed to the event handler either, which seems like an odd one to not pass to the event handler? So I am unsure of how to add a section to the script to send an email if the Problem is Acknowledged?

Finally, the standard XI notifications, when the service is in a Critical state resend the notification every hour it. However the event handler only fires off on the initial state change. I tried setting the service as volatile, which does force the event handler to run each time a check is conducted, but since we are polling for uptime every 5 minutes that will result in a ton of emails. Is there a way to have the event handler re-run every hour while the status is Critical? Or is it possible to specify a different check interval if the status is critical? So when Status is Ok it checks every 5 minutes, but once it's changed to critical it switches to every 60 minutes?

[vtrac]

Hi,
How are you doing?

Is there a way to have the event handler re-run every hour while the status is Critical?

I'm sorry, but there is not a way to set the event handler to re-ran every hour.


Regards,
Vinh