Nagios Event handler not working

[thanigaivel.a]

We have two environment, in one of the environment, we are using event handler without any issue for the last one year. However, we are deploying the same script in another environment, but its not working.

we have written this script to trigger ticket to our ticketing system, if a service or host goes down to "HARD & DOWN for host and CRITICAL & HARD" for service . (note: same script working well in another nagiosXI enterprise environement).

I've tried bring down a service using submit passive check and saw the logs, bizarrely event handler executing after the service turn to normal state and NOT after "CRITICAL & HARD" state.

so kindly help me to resolve the issue, IS ANY CONFIGURATION IS WRONG IN ANY CONFIG FILE TO MAKE EVENT HANDLER TO WORK WELL ?
Sharing the nagios.cfg and resource.cfg file and my event handler script.

Attached the screenshot which showing event handler triggering service turned to OK state.
######################### LOGS #####################################

[root@indeta3appu001 ~]# cat /usr/local/nagios/etc/nagios.cfg |grep event
log_event_handlers=1
global_host_event_handler=xi_host_event_handler
global_service_event_handler=xi_service_event_handler
enable_event_handlers=1
event_broker_options=-1
event_handler_timeout=30


[root@indeta3appu001 ~]# cat /usr/local/nagios/etc/resource.cfg
# Path to the plugins
$USER1$=/usr/local/nagios/libexec

# Path to event handlers
#$USER2$=/usr/local/nagios/libexec/eventhandlers

# Port for NSClient++
$USER7$=12489
# Password for NSClient++
$USER8$=secret

[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/libexec/eventhandlers/event_handler_service /usr/local/nagios/libexec/eventhandlers/event_handler_host
-rwxrwxrwx 1 nagios nagios 1388 Aug 3 20:14 /usr/local/nagios/libexec/eventhandlers/event_handler_host
-rwxrwxrwx 1 nagios nagios 1642 Aug 4 00:48 /usr/local/nagios/libexec/eventhandlers/event_handler_service

[thanigaivel.a]

Also, FOR YOUR INFORMATION I've tested event handler already as per the NagiosXI document by using "check_dummy". ITS WORKING PROPERLY.

Attached profile.

In the attached screenshot, its clearly shows that event handler script triggering after the state changes to OK.

[scottwilkerson]

the event handler points to

Code: Select all

$USER2$/event_handler_service

however
$USER2$ is not defined in resources.cfg

[thanigaivel.a]

Changed the resouce config. (see below).

Now again triggered the event handler by using passive submit, now again event handler script triggering after service state changes to OK only.

I dont know why its behaving like this, since I used to configure many event handler in our another environment, (Note: This is our new environment, triying to configure for the first time).

Attaching the screenshot of commands section.

################## LOGS ###################

[root@indeta3appu001 ~]# cat /usr/local/nagios/etc/resource.cfg
# Path to the plugins
$USER1$=/usr/local/nagios/libexec

# Path to event handlers
$USER2$=/usr/local/nagios/libexec/eventhandlers

# Port for NSClient++
$USER7$=12489
# Password for NSClient++
$USER8$=secret

[root@indeta3appu001 ~]#
[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/libexec/eventhandlers/event_handler_service
-rwxrwxrwx 1 nagios nagios 1642 Aug 4 00:48 /usr/local/nagios/libexec/eventhandlers/event_handler_service
[root@indeta3appu001 ~]#

[thanigaivel.a]

Sending the nagios.log. See its executing the script after service state changes to OK. And also attaching the screenshot of settings of that service.

[root@indeta3appu001 ~]# tail -f /var/log/messages |grep 13.61.25.230_gdcsdbpecs
Aug 4 02:33:06 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;test
Aug 4 02:33:06 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;xi_service_event_handler
Aug 4 02:33:06 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;event_handler_service
Aug 4 02:33:06 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;test
Aug 4 02:33:06 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;xi_service_event_handler
Aug 4 02:33:06 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;SOFT;1;event_handler_service
Aug 4 02:34:06 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;SWAP OK - 100% free (8063 MB out of 8063 MB)
Aug 4 02:34:06 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;xi_service_event_handler
Aug 4 02:34:06 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;event_handler_service
Aug 4 02:34:06 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;SWAP OK - 100% free (8063 MB out of 8063 MB)
Aug 4 02:34:06 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;xi_service_event_handler
Aug 4 02:34:06 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;SOFT;2;event_handler_service
Aug 4 02:34:06 indeta3appu001 nagios: wproc: stderr line 02: /bin/sh: -c: line 0: `/usr/local/nagios/libexec/eventhandlers/event_handler_service 13.61.25.230_gdcsdbpecs 13.61.25.230_gdcsdbpecs 13.61.25.230 Swap Usage Swap Usage OK SOFT 0 9227 20803 SWAP OK - 100% free (8063 MB out of 8063 MB) '
Aug 4 02:34:06 indeta3appu001 nagios: wproc: stderr line 02: /bin/sh: -c: line 0: `/usr/local/nagios/libexec/eventhandlers/event_handler_service 13.61.25.230_gdcsdbpecs 13.61.25.230_gdcsdbpecs 13.61.25.230 Swap Usage Swap Usage OK SOFT 0 9227 20803 SWAP OK - 100% free (8063 MB out of 8063 MB) '

[thanigaivel.a]

Now reduced the check Interval - 2 , Max check attenpt - 1. And tried to bring down the service by passive submit.

See the logs now. STILL SAME SCENARIO

[root@indeta3appu001 ~]# tail -f /var/log/messages |grep 13.61.25.230_gdcsdbpecs
Aug 4 02:39:36 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;test
Aug 4 02:39:36 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;xi_service_event_handler
Aug 4 02:39:36 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;test
Aug 4 02:39:36 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;xi_service_event_handler
Aug 4 02:39:36 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;event_handler_service
Aug 4 02:39:36 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;CRITICAL;HARD;1;event_handler_service
Aug 4 02:40:02 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;SWAP OK - 100% free (8063 MB out of 8063 MB)
Aug 4 02:40:02 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;xi_service_event_handler
Aug 4 02:40:02 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;event_handler_service
Aug 4 02:40:02 indeta3appu001 nagios: SERVICE ALERT: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;SWAP OK - 100% free (8063 MB out of 8063 MB)
Aug 4 02:40:02 indeta3appu001 nagios: GLOBAL SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;xi_service_event_handler
Aug 4 02:40:02 indeta3appu001 nagios: SERVICE EVENT HANDLER: 13.61.25.230_gdcsdbpecs;Swap Usage;OK;HARD;1;event_handler_service
Aug 4 02:40:02 indeta3appu001 nagios: wproc: stderr line 02: /bin/sh: -c: line 0: `/usr/local/nagios/libexec/eventhandlers/event_handler_service 13.61.25.230_gdcsdbpecs 13.61.25.230_gdcsdbpecs 13.61.25.230 Swap Usage Swap Usage OK HARD 0 9228 20806 SWAP OK - 100% free (8063 MB out of 8063 MB) '
Aug 4 02:40:02 indeta3appu001 nagios: wproc: stderr line 02: /bin/sh: -c: line 0: `/usr/local/nagios/libexec/eventhandlers/event_handler_service 13.61.25.230_gdcsdbpecs 13.61.25.230_gdcsdbpecs 13.61.25.230 Swap Usage Swap Usage OK HARD 0 9228 20806 SWAP OK - 100% free (8063 MB out of 8063 MB) '

[scottwilkerson]

Can you attach /usr/local/nagios/libexec/eventhandlers/event_handler_service ? it seems to be erroring in the file

[thanigaivel.a]

Code: Select all

[root@indeta3appu001 ~]# cat /usr/local/nagios/libexec/eventhandlers/event_handler_service
#!/bin/bash
rm -rf /tmp/bodynagios_service.json
DATE=$(date)
day=$(date +%A)
HOSTNAME=$1
HOSTALIAS=$2
HOSTADDRESS=$3
SERVICEDISPLAYNAME=$4
SERVICEDESC=$5
SERVICESTATE=$6
SERVICESTATETYPE=$7
SERVICEPROBLEMID=$8
LASTSERVICEPROBLEMID=$9
SERVICEEVENTID=$10
SERVICEOUTPUT=$11
LONGSERVICEOUTPUT=$12

data_service="{
    \"fields\": {
       \"project\":
       {
          \"key\": \"PDS\"
       },
       \"summary\": \" NagiosXI Service alert for $HOSTNAME: $SERVICEDISPLAYNAME is $SERVICESTATE\",
       \"description\": \"The Host $HOSTNAME:$HOSTADDRESS : $SERVICEDISPLAYNAME has changed to a $SERVICESTATE state at $DATE \",
       \"issuetype\": {
          \"name\": \"Incident\"
         },
\"customfield_10322\": {
            \"value\": \"Infrastructure-Nagios\" },
\"customfield_10207\": { \"value\": \"Normal\" },
        \"priority\": { \"id\": \"4\" }
   }
}"

if [ "$day" == "Saturday" ] || [ "$day" == "Sunday" ] ;then
exit
elif [[ "$SERVICESTATETYPE" == "HARD" && "$SERVICESTATE" == "CRITICAL" ]]; then

####if [[ "$SERVICESTATETYPE" == "HARD" && "$SERVICESTATE" == "CRITICAL" ]]; then

echo $data_service > /tmp/bodynagios_service.json

curl  -D- -k -u nagiosinteg:nagiosinteg -X POST -H "Content-Type: application/json" --data @/tmp/bodynagios_service.json http://testdev.sdsp.mc.xerox.com:8080/rest/api/2/issue/
exit 0
else
exit 0
fi

[thanigaivel.a]

Had posted the "/usr/local/nagios/libexec/eventhandlers/event_handler_service" script and below mentioned how I configured in commands section

################ Commands ########################

$USER2$/event_handler_service $HOSTNAME$ $HOSTALIAS$ $HOSTADDRESS$ $SERVICEDISPLAYNAME$ $SERVICEDESC$ $SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEPROBLEMID$ $LASTSERVICEPROBLEMID$ $SERVICEEVENTID$ $SERVICEOUTPUT$ $LONGSERVICEOUTPUT$

################ Resource file ####################

[root@indeta3appu001 ~]# cat /usr/local/nagios/etc/resource.cfg
# Path to the plugins
$USER1$=/usr/local/nagios/libexec

# Path to event handlers
$USER2$=

# Port for NSClient++
$USER7$=12489
# Password for NSClient++
$USER8$=secret

[root@indeta3appu001 ~]#


############################## Permissions for the directory given below" ######################

[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/
drwxr-xr-x. 9 root root 94 Mar 3 02:48 /usr/local/nagios/
[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/libexec/
drwsrwsr-x 3 nagios nagios 4096 Aug 4 01:25 /usr/local/nagios/libexec/
[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/libexec/eventhandlers/
drwxrwsrwx 2 apache nagios 124 Aug 6 20:46 /usr/local/nagios/libexec/eventhandlers/
[root@indeta3appu001 ~]# ls -ld /usr/local/nagios/libexec/eventhandlers/event_handler_service
-rwxrwxrwx 1 nagios nagios 1642 Aug 4 00:48 /usr/local/nagios/libexec/eventhandlers/event_handler_service
[root@indeta3appu001 ~]#

[scottwilkerson]

Can you run this from the CLI

Code: Select all

su nagios -c /usr/local/nagios/libexec/eventhandlers/event_handler_service 13.61.25.230_gdcsdbpecs 13.61.25.230_gdcsdbpecs 13.61.25.230 Swap Usage Swap Usage OK HARD 0 9228 20806'