Slack Integration for Nagios

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
sartecat
Posts: 33
Joined: Tue Jun 20, 2017 1:11 pm

Slack Integration for Nagios

Post by sartecat »

So I am following Matthew's instruction on how to integrate Nagios with slack.
http://matthewcmcmillan.blogspot.com/20 ... -with.html

So I followed the instructions...

#!/bin/bash

# This script is used by Nagios to post alerts into a Slack channel
# using the Incoming WebHooks integration. Create the channel, botname
# and integration first and then add this notification script in your
# Nagios configuration.
#
# All variables that start with NAGIOS_ are provided by Nagios as
# environment variables when an notification is generated.
# A list of the env variables is available here:
# http://nagios.sourceforge.net/docs/3_0/macrolist.html
#
# More info on Slack
# Website: https://slack.com/
# Twitter: @slackhq, @slackapi
#
# My info
# Website: http://matthewcmcmillan.blogspot.com/
# Twitter: @matthewmcmillan

#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="[TEAMNAME].slack.com"
SLACK_TOKEN="[SLACKKEY]"
SLACK_CHANNEL="#nagios-alertyourslacks"
SLACK_BOTNAME="nagios"


#Set the message icon based on Nagios service state
if [ "$NAGIOS_SERVICESTATE" = "CRITICAL" ]
then
ICON=":exclamation:"
elif [ "$NAGIOS_SERVICESTATE" = "WARNING" ]
then
ICON=":warning:"
elif [ "$NAGIOS_SERVICESTATE" = "OK" ]
then
ICON=":white_check_mark:"
elif [ "$NAGIOS_SERVICESTATE" = "UNKNOWN" ]
then
ICON=":question:"
else
ICON=":white_medium_square:"
fi

#Send message to Slack
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}

My commands.cfg
# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}

# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}


My contacts.cfg:
define contact {
contact_name slack
alias Slack
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-slack
host_notification_commands notify-host-by-slack
}

define contactgroup{
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin,slack
}

define contactgroup{
contactgroup_name admins-page
alias Nagios Administrators
members nagiosadmin,slack
}


This is what my Notifications look like on Nagios:
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 slack notify-service-by-slack CRITICAL - Plugin timed out while executing system call
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 nagiosadmin notify-service-by-email CRITICAL - Plugin timed out while executing system call
paloalto PING CRITICAL 07-07-2017 10:34:49 slack notify-service-by-slack CRITICAL - Host Unreachable (172.16.0.6)
paloalto PING CRITICAL 07-07-2017 10:34:49 nagiosadmin notify-service-by-email CRITICAL - Host Unreachable (172.16.0.6)

I received the email notification, but didn't get any for slack. Any ideas?
Last edited by sartecat on Mon Jul 10, 2017 1:53 pm, edited 2 times in total.
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Slack Integration for Nagios

Post by mcapra »

Can you share the contents of /tmp/slack.log? If that file doesn't exist, you might want to create such a file and make sure the Nagios system user can write to that file, since your Nagios command definitions for notify-service/host-by-slack are redirecting the script's output to that file.
Former Nagios employee
https://www.mcapra.com/
scottwilkerson
DevOps Engineer
Posts: 19396
Joined: Tue Nov 15, 2011 3:11 pm
Location: Nagios Enterprises
Contact:

Re: Slack Integration for Nagios

Post by scottwilkerson »

Also can you send the output of

Code: Select all

ls -l /usr/local/bin/slack_nagios.sh
ls -l /tmp/slack.log
and I am going to assume you modified these already right?

Code: Select all

#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="[TEAMNAME].slack.com"
SLACK_TOKEN="[SLACKKEY]"
SLACK_CHANNEL="#nagios-alertyourslacks"
SLACK_BOTNAME="nagios"
Former Nagios employee
Creator:
ahumandesign.com
enneagrams.com
sartecat
Posts: 33
Joined: Tue Jun 20, 2017 1:11 pm

Re: Slack Integration for Nagios

Post by sartecat »

Thanks for the fast reply.

When I issued the following:

Code: Select all

# cat /tmp/slack.log
/bin/sh: /usr/local/bin/slack_nagios.sh: Permission denied

Code: Select all

# ls -l /usr/local/bin/slack
-rw-r--r-- 1 root root 1718 Jul  7 10:27 /usr/local/bin/slack_nagios.sh

Code: Select all

# ls -l /tmp/slack.log
-rw-r--r-- 1 nagios nagios 59 Jul  7 10:43 /tmp/slack.log
and I modified those variables
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Slack Integration for Nagios

Post by tgriep »

The script needs to be executable as well.
Login to the server as root and run the following to change that.

Code: Select all

chmod a+x /usr/local/bin/slack_nagios.sh
Then see if it can be run by the Nagios Daemon after the change.
Be sure to check out our Knowledgebase for helpful articles and solutions!
sartecat
Posts: 33
Joined: Tue Jun 20, 2017 1:11 pm

Re: Slack Integration for Nagios

Post by sartecat »

I'm not sure what token I should be using as well. It said to use the incoming webhooks integration, and botname called nagios. I have the custom integration Bots and Incoming WebHooks configured. I'm not entirely sure what to use for that.
sartecat
Posts: 33
Joined: Tue Jun 20, 2017 1:11 pm

Re: Slack Integration for Nagios

Post by sartecat »

I think I figured out the token. But all I'm getting on slack is this message:
2:24 :white_medium_square: HOST: SERVICE: MESSAGE: See Nagios

The script has
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}

It's not pulling the NAGIOS_HOSTNAME and NAGIOS_SERVICEDISPLAYNAME...etc.
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: Slack Integration for Nagios

Post by mcapra »

Hmm, I think the script you're using needs to be modified to accept those Nagios based macros. Here's a big old list of all the available Nagios macros and their contexts:
https://assets.nagios.com/downloads/nag ... olist.html

The way you would pass those macros from Nagios off to the Slack script is via the Nagios command definition (notify-host-by-slack and notify-service-by-slack). Here's one of the stock Nagios notification command definitions:

Code: Select all

define command{
        command_name    notify-service-by-email
        command_line    /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
        }
You may notice a lot of $MACRO$ definitions in that command_line parameter. This is basically how Nagios is able to get "Nagios related information" from Nagios to the external script of your choice. Nagios will, on the back-end, substitute those macro values at run-time for the script.

You may also need to modify the href it's trying to stuff into the Slack message, unless your path actually is /nagios3:

Code: Select all

<https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>
Former Nagios employee
https://www.mcapra.com/
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: Slack Integration for Nagios

Post by tgriep »

Another thing to look at is if you do not create the command in Nagios, passing all of the macros and are going to replay on the environment variables for that, you would have to enable it in the nagios.cfg file by adding the following option and restarting the nagios processes.

Code: Select all

enable_environment_macros=1
Try that and see if they get passed to your script.
Environment Macros Option

Format: enable_environment_macros=<0/1>
Example: enable_environment_macros=0

This option determines whether or not the Nagios daemon will make all standard macros available as environment variables to your check, notification, event hander, etc. commands. In large Nagios installations this can be problematic because it takes additional memory and (more importantly) CPU to compute the values of all macros and make them available to the environment.

0 = Don't make macros available as environment variables
1 = Make macros available as environment variables (default)
Be sure to check out our Knowledgebase for helpful articles and solutions!
sartecat
Posts: 33
Joined: Tue Jun 20, 2017 1:11 pm

Re: Slack Integration for Nagios

Post by sartecat »

This was not included in the integration guide! Thank you, it works!
tgriep wrote:Another thing to look at is if you do not create the command in Nagios, passing all of the macros and are going to replay on the environment variables for that, you would have to enable it in the nagios.cfg file by adding the following option and restarting the nagios processes.

Code: Select all

enable_environment_macros=1
Try that and see if they get passed to your script.
Environment Macros Option

Format: enable_environment_macros=<0/1>
Example: enable_environment_macros=0

This option determines whether or not the Nagios daemon will make all standard macros available as environment variables to your check, notification, event hander, etc. commands. In large Nagios installations this can be problematic because it takes additional memory and (more importantly) CPU to compute the values of all macros and make them available to the environment.

0 = Don't make macros available as environment variables
1 = Make macros available as environment variables (default)
Locked