Slack Integration for Nagios
Slack Integration for Nagios
So I am following Matthew's instruction on how to integrate Nagios with slack.
http://matthewcmcmillan.blogspot.com/20 ... -with.html
So I followed the instructions...
#!/bin/bash
# This script is used by Nagios to post alerts into a Slack channel
# using the Incoming WebHooks integration. Create the channel, botname
# and integration first and then add this notification script in your
# Nagios configuration.
#
# All variables that start with NAGIOS_ are provided by Nagios as
# environment variables when an notification is generated.
# A list of the env variables is available here:
# http://nagios.sourceforge.net/docs/3_0/macrolist.html
#
# More info on Slack
# Website: https://slack.com/
# Twitter: @slackhq, @slackapi
#
# My info
# Website: http://matthewcmcmillan.blogspot.com/
# Twitter: @matthewmcmillan
#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="[TEAMNAME].slack.com"
SLACK_TOKEN="[SLACKKEY]"
SLACK_CHANNEL="#nagios-alertyourslacks"
SLACK_BOTNAME="nagios"
#Set the message icon based on Nagios service state
if [ "$NAGIOS_SERVICESTATE" = "CRITICAL" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "WARNING" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "OK" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "UNKNOWN" ]
then
ICON=""
else
ICON=""
fi
#Send message to Slack
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
My commands.cfg
# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
My contacts.cfg:
define contact {
contact_name slack
alias Slack
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-slack
host_notification_commands notify-host-by-slack
}
define contactgroup{
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin,slack
}
define contactgroup{
contactgroup_name admins-page
alias Nagios Administrators
members nagiosadmin,slack
}
This is what my Notifications look like on Nagios:
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 slack notify-service-by-slack CRITICAL - Plugin timed out while executing system call
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 nagiosadmin notify-service-by-email CRITICAL - Plugin timed out while executing system call
paloalto PING CRITICAL 07-07-2017 10:34:49 slack notify-service-by-slack CRITICAL - Host Unreachable (172.16.0.6)
paloalto PING CRITICAL 07-07-2017 10:34:49 nagiosadmin notify-service-by-email CRITICAL - Host Unreachable (172.16.0.6)
I received the email notification, but didn't get any for slack. Any ideas?
http://matthewcmcmillan.blogspot.com/20 ... -with.html
So I followed the instructions...
#!/bin/bash
# This script is used by Nagios to post alerts into a Slack channel
# using the Incoming WebHooks integration. Create the channel, botname
# and integration first and then add this notification script in your
# Nagios configuration.
#
# All variables that start with NAGIOS_ are provided by Nagios as
# environment variables when an notification is generated.
# A list of the env variables is available here:
# http://nagios.sourceforge.net/docs/3_0/macrolist.html
#
# More info on Slack
# Website: https://slack.com/
# Twitter: @slackhq, @slackapi
#
# My info
# Website: http://matthewcmcmillan.blogspot.com/
# Twitter: @matthewmcmillan
#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="[TEAMNAME].slack.com"
SLACK_TOKEN="[SLACKKEY]"
SLACK_CHANNEL="#nagios-alertyourslacks"
SLACK_BOTNAME="nagios"
#Set the message icon based on Nagios service state
if [ "$NAGIOS_SERVICESTATE" = "CRITICAL" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "WARNING" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "OK" ]
then
ICON=""
elif [ "$NAGIOS_SERVICESTATE" = "UNKNOWN" ]
then
ICON=""
else
ICON=""
fi
#Send message to Slack
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
My commands.cfg
# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/local/bin/slack_nagios.sh > /tmp/slack.log 2>&1
}
My contacts.cfg:
define contact {
contact_name slack
alias Slack
service_notification_period 24x7
host_notification_period 24x7
service_notification_options w,u,c,r
host_notification_options d,r
service_notification_commands notify-service-by-slack
host_notification_commands notify-host-by-slack
}
define contactgroup{
contactgroup_name admins
alias Nagios Administrators
members nagiosadmin,slack
}
define contactgroup{
contactgroup_name admins-page
alias Nagios Administrators
members nagiosadmin,slack
}
This is what my Notifications look like on Nagios:
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 slack notify-service-by-slack CRITICAL - Plugin timed out while executing system call
paloalto MGT Port Link Status CRITICAL 07-07-2017 10:36:11 nagiosadmin notify-service-by-email CRITICAL - Plugin timed out while executing system call
paloalto PING CRITICAL 07-07-2017 10:34:49 slack notify-service-by-slack CRITICAL - Host Unreachable (172.16.0.6)
paloalto PING CRITICAL 07-07-2017 10:34:49 nagiosadmin notify-service-by-email CRITICAL - Host Unreachable (172.16.0.6)
I received the email notification, but didn't get any for slack. Any ideas?
Last edited by sartecat on Mon Jul 10, 2017 1:53 pm, edited 2 times in total.
Re: Slack Integration for Nagios
Can you share the contents of /tmp/slack.log? If that file doesn't exist, you might want to create such a file and make sure the Nagios system user can write to that file, since your Nagios command definitions for notify-service/host-by-slack are redirecting the script's output to that file.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Slack Integration for Nagios
Also can you send the output of
and I am going to assume you modified these already right?
Code: Select all
ls -l /usr/local/bin/slack_nagios.sh
ls -l /tmp/slack.log
Code: Select all
#Modify these variables for your environment
MY_NAGIOS_HOSTNAME="nagios.yourdomain.com"
SLACK_HOSTNAME="[TEAMNAME].slack.com"
SLACK_TOKEN="[SLACKKEY]"
SLACK_CHANNEL="#nagios-alertyourslacks"
SLACK_BOTNAME="nagios"
Re: Slack Integration for Nagios
Thanks for the fast reply.
When I issued the following:
and I modified those variables
When I issued the following:
Code: Select all
# cat /tmp/slack.log
/bin/sh: /usr/local/bin/slack_nagios.sh: Permission denied
Code: Select all
# ls -l /usr/local/bin/slack
-rw-r--r-- 1 root root 1718 Jul 7 10:27 /usr/local/bin/slack_nagios.sh
Code: Select all
# ls -l /tmp/slack.log
-rw-r--r-- 1 nagios nagios 59 Jul 7 10:43 /tmp/slack.log
Re: Slack Integration for Nagios
The script needs to be executable as well.
Login to the server as root and run the following to change that.
Then see if it can be run by the Nagios Daemon after the change.
Login to the server as root and run the following to change that.
Code: Select all
chmod a+x /usr/local/bin/slack_nagios.sh
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Slack Integration for Nagios
I'm not sure what token I should be using as well. It said to use the incoming webhooks integration, and botname called nagios. I have the custom integration Bots and Incoming WebHooks configured. I'm not entirely sure what to use for that.
Re: Slack Integration for Nagios
I think I figured out the token. But all I'm getting on slack is this message:
2:24 HOST: SERVICE: MESSAGE: See Nagios
The script has
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
It's not pulling the NAGIOS_HOSTNAME and NAGIOS_SERVICEDISPLAYNAME...etc.
2:24 HOST: SERVICE: MESSAGE: See Nagios
The script has
curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"text\": \"${ICON} HOST: ${NAGIOS_HOSTNAME} SERVICE: ${NAGIOS_SERVICEDISPLAYNAME} MESSAGE: ${NAGIOS_SERVICEOUTPUT} <https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>\"}" https://${SLACK_HOSTNAME}/services/hooks/incoming-webhook?token=${SLACK_TOKEN}
It's not pulling the NAGIOS_HOSTNAME and NAGIOS_SERVICEDISPLAYNAME...etc.
Re: Slack Integration for Nagios
Hmm, I think the script you're using needs to be modified to accept those Nagios based macros. Here's a big old list of all the available Nagios macros and their contexts:
https://assets.nagios.com/downloads/nag ... olist.html
The way you would pass those macros from Nagios off to the Slack script is via the Nagios command definition (notify-host-by-slack and notify-service-by-slack). Here's one of the stock Nagios notification command definitions:
You may notice a lot of $MACRO$ definitions in that command_line parameter. This is basically how Nagios is able to get "Nagios related information" from Nagios to the external script of your choice. Nagios will, on the back-end, substitute those macro values at run-time for the script.
You may also need to modify the href it's trying to stuff into the Slack message, unless your path actually is /nagios3:
https://assets.nagios.com/downloads/nag ... olist.html
The way you would pass those macros from Nagios off to the Slack script is via the Nagios command definition (notify-host-by-slack and notify-service-by-slack). Here's one of the stock Nagios notification command definitions:
Code: Select all
define command{
command_name notify-service-by-email
command_line /usr/bin/printf "%b" "***** Nagios *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}
You may also need to modify the href it's trying to stuff into the Slack message, unless your path actually is /nagios3:
Code: Select all
<https://${MY_NAGIOS_HOSTNAME}/cgi-bin/nagios3/status.cgi?host=${NAGIOS_HOSTNAME}|See Nagios>
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Slack Integration for Nagios
Another thing to look at is if you do not create the command in Nagios, passing all of the macros and are going to replay on the environment variables for that, you would have to enable it in the nagios.cfg file by adding the following option and restarting the nagios processes.
Try that and see if they get passed to your script.
Code: Select all
enable_environment_macros=1
Environment Macros Option
Format: enable_environment_macros=<0/1>
Example: enable_environment_macros=0
This option determines whether or not the Nagios daemon will make all standard macros available as environment variables to your check, notification, event hander, etc. commands. In large Nagios installations this can be problematic because it takes additional memory and (more importantly) CPU to compute the values of all macros and make them available to the environment.
0 = Don't make macros available as environment variables
1 = Make macros available as environment variables (default)
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Slack Integration for Nagios
This was not included in the integration guide! Thank you, it works!
tgriep wrote:Another thing to look at is if you do not create the command in Nagios, passing all of the macros and are going to replay on the environment variables for that, you would have to enable it in the nagios.cfg file by adding the following option and restarting the nagios processes.Try that and see if they get passed to your script.Code: Select all
enable_environment_macros=1
Environment Macros Option
Format: enable_environment_macros=<0/1>
Example: enable_environment_macros=0
This option determines whether or not the Nagios daemon will make all standard macros available as environment variables to your check, notification, event hander, etc. commands. In large Nagios installations this can be problematic because it takes additional memory and (more importantly) CPU to compute the values of all macros and make them available to the environment.
0 = Don't make macros available as environment variables
1 = Make macros available as environment variables (default)