Then, run the following commands and post the output.
Code: Select all
su - nagios
ls -l /tmp
ls -l /usr/local/bin
/usr/local/bin/slack_nagios.sh
Code: Select all
su - nagios
ls -l /tmp
ls -l /usr/local/bin
/usr/local/bin/slack_nagios.sh
Thanks, tgriep...tgriep wrote:Can you post how you test the command from the command line so I can see the steps you are doing to run it?
Then, run the following commands and post the output.ThanksCode: Select all
su - nagios ls -l /tmp ls -l /usr/local/bin /usr/local/bin/slack_nagios.sh
Code: Select all
root@monitor:/usr/local/nagios/var# su - nagios
nagios@monitor:~$ ls -l /tmp
total 2520
drwxrwxr-x 17 root root 4096 Jun 12 05:47 nagios-plugins-release-2.2.1
-rw-r--r-- 1 root root 2049050 Jun 12 05:44 nagios-plugins.tar.gz
drwxrwxr-x 10 root root 4096 Jun 12 05:52 nrpe-nrpe-3.1.1
-rw-r--r-- 1 root root 515243 Jun 12 05:49 nrpe.tar.gz
-rwxrwxrwx 1 nagios nagios 0 Aug 1 05:56 slack.log
-rw-rw-rw- 1 root root 0 Aug 1 17:21 slackapi.log
drwx------ 2 pilotmc staff 4096 Aug 1 16:03 ssh-CW1Ff9uoTk
nagios@monitor:~$ ls -l /usr/local/bin
total 20
-rw-r--r-- 1 nagios nagios 1441 Aug 1 17:32 slack_nagios.php
-rwxr-xr-x 1 root staff 1654 May 23 21:00 slack_nagios.sh-ORIG
-rwxr-xr-x 1 nagios nagios 1455 Jun 22 21:35 slack_nagios2.sh
-rwxr-xr-x 1 root staff 78 Jun 22 18:06 test.php
-rwxr-xr-x 1 pilotmc staff 293 May 23 05:55 test_alert.sh
nagios@monitor:~$ /usr/bin/php /usr/local/bin/slack_nagios.php test test OK test test
nagios@monitor:~$
Code: Select all
# 'notify-service-by-slack' command definition
define command {
command_name notify-service-by-slack
command_line /usr/bin/php /usr/local/bin/slack_nagios.php $HOSTNAME$ $SERVICEDESC$ $SERVICESTATE$ "$SERVICEOUTPUT$" $NOTIFICATIONTYPE$
}
# 'notify-host-by-slack' command definition
define command {
command_name notify-host-by-slack
command_line /usr/bin/php /usr/local/bin/slack_nagios.php $HOSTNAME$ $SERVICEDESC$ $SERVICESTATE$ "$SERVICEOUTPUT$" $NOTIFICATIONTYPE$
}
Code: Select all
[1501694687.405609] [032.2] [pid=27309] ** Notifying contact 'slack'
[1501694687.405619] [032.2] [pid=27309] Raw notification command: /usr/bin/php /usr/local/bin/slack_nagios.php $HOSTNAME$ $SERVICEDESC$ $SERVICESTATE$ "$SERVICEOUTPUT$" $NOTIFICATIONTYPE$
[1501694687.405646] [032.2] [pid=27309] Processed notification command: /usr/bin/php /usr/local/bin/slack_nagios.php staff Test CRITICAL "(1) YOWZA" PROBLEM
[1501694687.405774] [032.2] [pid=27309] Calculating next valid notification time...
[1501694687.405782] [032.2] [pid=27309] Default interval: 10.000000
[1501694687.405791] [032.2] [pid=27309] Interval used for calculating next valid notification time: 10.000000
[1501694687.405802] [032.0] [pid=27309] 3 contacts were notified. Next possible notification time: Wed Aug 2 17:34:47 2017
[1501694687.405808] [032.0] [pid=27309] 3 contacts were notified.
Code: Select all
[1501694687] SERVICE ALERT: staff;Test;CRITICAL;HARD;1;(1) < YOWZA
[1501694687] SERVICE NOTIFICATION: opsteam;staff;Test;CRITICAL;notify-service-by-email;(1) < YOWZA
[1501694687] SERVICE NOTIFICATION: admins;staff;Test;CRITICAL;notify-service-by-email;(1) < YOWZA
[1501694687] SERVICE NOTIFICATION: slack;staff;Test;CRITICAL;notify-service-by-slack;(1) < YOWZA
[1501694717] wproc: Core Worker 27312: job 2242 (pid=6403) timed out. Killing it
[1501694717] wproc: NOTIFY job 2242 from worker Core Worker 27312 timed out after 30.03s
[1501694717] wproc: host=staff; service=Test; contact=slack
[1501694717] wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1501694717] Warning: Notifying contact 'slack' of service 'Test' on host 'staff' by command '/usr/bin/php /usr/local/bin/slack_nagios.php staff Test CRITICAL "(1) YOWZA" PROBLEM' timed out after 0.00 seconds
[1501694717] wproc: Core Worker 27312: job 2242 (pid=6403): Dormant child reaped
[1501694747] SERVICE ALERT: staff;Test;OK;HARD;1;Log check ok - 0 pattern matches found
[1501694747] SERVICE NOTIFICATION: opsteam;staff;Test;OK;notify-service-by-email;Log check ok - 0 pattern matches found
[1501694747] SERVICE NOTIFICATION: admins;staff;Test;OK;notify-service-by-email;Log check ok - 0 pattern matches found
[1501694747] SERVICE NOTIFICATION: slack;staff;Test;OK;notify-service-by-slack;Log check ok - 0 pattern matches found
[1501694777] wproc: Core Worker 27312: job 2246 (pid=6494) timed out. Killing it
[1501694777] wproc: NOTIFY job 2246 from worker Core Worker 27312 timed out after 30.03s
[1501694777] wproc: host=staff; service=Test; contact=slack
[1501694777] wproc: early_timeout=1; exited_ok=0; wait_status=0; error_code=62;
[1501694777] Warning: Notifying contact 'slack' of service 'Test' on host 'staff' by command '/usr/bin/php /usr/local/bin/slack_nagios.php staff Test OK "Log check ok - 0 pattern matches found" RECOVERY' timed out after 0.00 seconds
[1501694777] wproc: Core Worker 27312: job 2246 (pid=6494): Dormant child reaped
Code: Select all
<?php
// Nagios-Slack Interface
//
// Sends alerts to Slack using the Slack API
//
/* Variables from the notification command:
NAGIOS_HOSTNAME=$1
NAGIOS_SERVICEDISPLAYNAME=$2
NAGIOS_SERVICESTATE=$3
NAGIOS_SERVICEOUTPUT=$4
NAGIOS_NOTIFICATIONYTYPE=$5
*/
$server="monitor.mydomain.com";
$token="*******/*******/************************";
$channel="#alerts";
$botname="Nagios";
if( $argv[3] == "OK" )
$icon = ":white_check_mark:";
else if( $argv[3] == "WARNING" )
$icon = ":warning:";
else if( $argv[3] == "CRITICAL" )
$icon = ":error:";
else
$icon = ":grey_question:";
$payload = array();
$payload['channel'] = $channel;
$payload['username'] = $botname;
$payload['icon_emoji'] = $icon;
$payload['text'] = sprintf( "HOST: %s\nSERVICE: %s\nMESSAGE: %s. See <https://%s/nagios/cgi-bin/extinfo.cgi?type=1&host=%s|See Nagios>", $argv[1], $argv[2], $argv[4], $server, $argv[1] );
$post = json_encode($payload, JSON_UNESCAPED_SLASHES );
$ch = curl_init("https://hooks.slack.com/services/$token");
curl_setopt( $ch, CURLOPT_POSTFIELDS, $post );
curl_setopt( $ch, CURLOPT_POST, 1 );
curl_setopt( $ch, CURLOPT_HTTPHEADER, array("Content-type: application/json") );
curl_setopt( $ch, CURLOPT_SSL_VERIFYHOST, 0 );
curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, 0 );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
$result = curl_exec( $ch );
syslog( LOG_INFO, "slack-nagios: {$payload['text']}");
?>
Do you have the historical logs to grep through? Knowing when the issue started could help tie it to an upgrade or permissions issue.pilotmc wrote:it just stopped working at some point