integration with Slack

[iwang]

hello,

I am trying to associate the nagios alerts with Slack. I referred this doc:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

but i got 404 respond. can you help take a look?

Code: Select all

-bash-4.1# ./slack_nagios.pl -field slack_channel=#dvmt30load -field HOSTALIAS="Test Host" -field HOSTSTATE=“Warning” -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE=“LOADISSUE”
Request:
POST https://xxxxxx.slack.com/services/hooks/nagios?token=xxxxxxxx
Content-Length: 165
Content-Type: application/x-www-form-urlencoded

HOSTOUTPUT=Host+is+UP&NOTIFICATIONTYPE=%E2%80%9CLOADISSUE%E2%80%9D&slack_version=1.1&slack_channel=dvmt30load&HOSTSTATE=%E2%80%9CWarning%E2%80%9D&HOSTALIAS=Test+Host

Response:
HTTP/1.1 404 Not Found
Connection: close
Date: Mon, 09 Mar 2020 22:22:31 GMT
Server: Cisco Umbrella
Content-Type: text/html
Client-Date: Mon, 09 Mar 2020 22:22:31 GMT
Client-Peer: 146.112.61.106:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /O=Cisco/CN=Cisco Umbrella Secondary SubCA pao-SG
Client-SSL-Cert-Subject: /C=US/ST=California/L=San Francisco/O=OpenDNS, Inc./CN=*.opendns.com
Client-SSL-Cipher: AES256-GCM-SHA384
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked

[jdunitz]

This may not be all of your problem, but I did notice that you have matching quotes rather than regular double quotes around at least one of your variables. That's going to confuse things.

I ran it through octal dump to make sure it was what I thought it was:

Code: Select all

$  head -1 1.txt | od -c
0000000 342 200 234   L   O   A   D   I   S   S   U   E 342 200 235  \n
0000020

 
$  tail -1 1.txt | od -c
0000000   "   L   O   A   D   I   S   S   U   E   "  \n
0000014

So, try using regular double quotes ("like this") and see if that helps. If not, let us know and we'll look for the next part of the issue.

Thanks!

--Jeffrey

[iwang]

thanks. somehow the quotes might not be the only reason. i had corrected the quotes but still got the 404. thank you

Code: Select all

./slack_nagios.pl -field slack_channel=#dvmt30load -field HOSTALIAS="Test Host" -field HOSTSTATE="Warning" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="LOADISSUE"

Request:
POST https://xxxxslack.com/services/hooks/nagios?token=xxxxxx
Content-Length: 132
Content-Type: application/x-www-form-urlencoded

HOSTOUTPUT=Host+is+UP&NOTIFICATIONTYPE=LOADISSUE&slack_version=1.1&slack_channel=%23dvmt30load&HOSTSTATE=Warning&HOSTALIAS=Test+Host

Response:
HTTP/1.1 404 Not Found
Connection: close
Date: Wed, 11 Mar 2020 00:21:21 GMT
Server: Cisco Umbrella
Content-Type: text/html
Client-Date: Wed, 11 Mar 2020 00:21:21 GMT
Client-Peer: 146.112.61.106:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /O=Cisco/CN=Cisco Umbrella Secondary SubCA pao-SG
Client-SSL-Cert-Subject: /C=US/ST=California/L=San Francisco/O=OpenDNS, Inc./CN=*.opendns.com
Client-SSL-Cipher: AES256-GCM-SHA384
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked

[jdunitz]

I don't think that notification type is correct.

Can you try your command like so:

Code: Select all

./slack_nagios.pl -field slack_channel=#dvmt30load -field HOSTALIAS="Test Host" -field HOSTSTATE="UP" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="RECOVERY"

and see if that works?

[iwang]

thanks. I tried. but no luck, same error.

even the previous notificationtype was directly copied from the nagios reference link: https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[iwang]

hello, eventually i decded to use webhook instead of the Nagios slack app.

i was able to install the commandline, service, but somehow the service status always in pending status.

i had referenced this tool: https://github.com/RunlevelConsulting/N ... ifications
and created service and command as below:

Code: Select all

define command {
    command_name    notfify-slack-techops-channel
    command_line    $USER1$/notification_slack.sh -a "$NOTIFICATIONTYPE$" -b $HOSTNAME$ -c "$HOSTSTATE$" -d "$HOSTOUTPUT$" -y "techops-alerts" -z "xxxxxx"
}

define service {
    host_name              xxxx.dvms.local
    service_description    notify slack techops channel
    display_name           notify slack techops channel
    check_command          notfify-slack-techops-channel!!!!!!!!
    max_check_attempts     5
    check_interval         2
    retry_interval         1
    check_period           workhours
    register               1
}

is there anything wrong what i had done? please check attached screenshot.

I don't think that notification type is correct.

Can you try your command like so:

Code: Select all

./slack_nagios.pl -field slack_channel=#dvmt30load -field HOSTALIAS="Test Host" -field HOSTSTATE="UP" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="RECOVERY"

and see if that works?

[jdunitz]

It's hard to say why you're getting the result you're getting, but let's investigate some more.


In the notification_slack.sh script, there are a few configuration lines that you'll have to edit to match your local setup:

SLACK_HOSTNAME="slack_hostname.slack.com"
SLACK_BOTUSERNAME="nagios-alerts"
MONITORING_URL="https://nagios_server.domain/nagios"
SLACK_CHANNEL="#${CHANNEL}"
WEBHOOK_ADDRESS="https://hooks.slack.com/services/${WEBHOOK_PATH}"

You made sure to set these to the correct values, yes?

If so, can you test the ./notification_slack.sh script on the command line, and does it work?

Also--and this applies to getting both plugins to work--be sure the slack channel you're pointing to exists, and that you can reach the slack servers from your nagios machine.

Hope this helps!

Finally, I recommend that, at least until you get things working, you set your notification time to 24x7, rather than workhours. It shouldn't make a difference, but if there's something strange with how your times are set up, we want to make sure that's not getting in the way.

[iwang]

thank you for your help.
I was able to leverage the script for slack integration. but.... i have to define one webhook for each slack channel. which the IT limited. so i have to stick to the previous perl solution

here is the output after I updated the openssl and curl:

Code: Select all

-bash-4.1# ./slack_nagios.pl -field slack_channel=#adfadfasdfdsa -field HOSTALIAS="Test Host" -field HOSTSTATE="UP" -field HOSTOUTPUT="Host is UP" -field NOTIFICATIONTYPE="RECOVERY"
Request:
POST https://isi-eng.slack.com/services/hooks/nagios?token=ssdfdsfdsxxxx
Content-Length: 126
Content-Type: application/x-www-form-urlencoded

HOSTOUTPUT=Host+is+UP&NOTIFICATIONTYPE=RECOVERY&slack_version=1.1&slack_channel=%23dvmt30load&HOSTSTATE=UP&HOSTALIAS=Test+Host

SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv3 write client hello A
SSL3 alert read:fatal:handshake failure
SSL_connect:failed in SSLv3 read server hello A
SSL_connect:before/connect initialization
SSL_connect:SSLv2 write client hello A
SSL_connect:failed in SSLv2 read server hello A
Response:
500 SSL negotiation failed: 
Content-Type: text/plain
Client-Date: Fri, 13 Mar 2020 22:46:25 GMT
Client-Warning: Internal response

500 SSL negotiation failed: 

You have new mail in /var/spool/mail/root

I have tried disable to use SSLv2/v3 but TLS1.2, but no luck:

Code: Select all

use warnings;
use strict;

use Getopt::Long;
use HTTP::Request::Common qw(POST);
use HTTP::Status qw(is_client_error);
use LWP::UserAgent;

$ENV{HTTPS_DEBUG} = 1;

#
# Customizable vars. Set these to the information for your team
#

my $opt_domain = "isi-eng.slack.com"; # Your team's domain
my $opt_token = "gDzxcp10OfkH5AyX4U64ywxx"; # The token from your Nagios services page


#
# Get command-line opts
#

my %opt_fields;
GetOptions("field=s%" => \%opt_fields);


#
# DO THINGS
#

my %event;

# Get all Nagios variables
while ((my $k, my $v) = each %ENV) {
	next unless $k =~ /^(?:NAGIOS|ICINGA)_(.*)$/;
	$event{$1} = $v;
}

# Merge in passed-in variables
%event = (%event, %opt_fields);

$event{"slack_version"} = "1.1";


#
# Make the request
#

#my $ua = LWP::UserAgent->new;

my $ua = LWP::UserAgent->new(ssl_opts => {
    verify_hostname => 0,
    SSL_version => 'TLSv1_2',
});

$ua->timeout(15);

my $req = POST("https://${opt_domain}/services/hooks/nagios?token=${opt_token}", \%event);

my $s = $req->as_string;
print STDERR "Request:\n$s\n";

my $resp = $ua->request($req);
$s = $resp->as_string;
print STDERR "Response:\n$s\n";

I also tried:
SSL_version => 'TLSv1_2',
SSL_version => 'TLSv12:!SSLv2:!SSLv3:!TLSv1',

[jdunitz]

Are you using CentOS or RHEL version 6 or 7?

I ask, because Slack just started enforcing SNI, and that may be an issue for the older versions of perl that come with those.

Let us know what you've got.

Thanks!

[iwang]

you are right. it's CentOS 6. it's caused by the old version of perl.
after i switched to use python, i am able to have the slack integration. it tested successfully on nagios server side.

even though, i had defined the commands, the contacts configuration defined, it doesn't push notification to the channel though. what i missed here?

for some commands, i imported from CCM, but when creating a contact, the command cannot be found from the list, why?

one more question on the contact configuration:
we have services associated to different hosts, for example service S applied to host h1, h2... h3. the notification config on the service S end will apply all hosts? we need for S on h1 send to channel teamA, but S issue on h2 send to teamB send to channel B. how to implement it?

Code: Select all

define contact {
    contact_name slack_techops
    alias slack_techops
    service_notification_period 24x7
    host_notification_period 24x7
    service_notification_options w,u,c,r
    host_notification_options d,r
    host_notification_commands slack_python_host_techops_c
    service_notification_commands slack_python_service_techops_c
}


define command {
    command_name    slack_python_host_techops_c
    command_line     $USER1$/slack2.py  -c techops-alerts -field HOSTALIAS=$HOSTNAME$ -field HOSTSTATE="$HOSTSTATE$" -field HOSTOUTPUT="$HOSTOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}


define command {
    command_name    slack_python_service_techops_c
    command_line    $USER1$/slack2.py  -c techops-alerts -field HOSTALIAS="$HOSTNAME$" -field SERVICEDESC="$SERVICEDESC$" -field SERVICESTATE="$SERVICESTATE$" -field SERVICEOUTPUT="$SERVICEOUTPUT$" -field NOTIFICATIONTYPE="$NOTIFICATIONTYPE$"
}

Are you using CentOS or RHEL version 6 or 7?

I ask, because Slack just started enforcing SNI, and that may be an issue for the older versions of perl that come with those.

Let us know what you've got.

Thanks!