Command works on CLI but not in SNMPTT event handler

[gavnor]

Using the Nagios XI interface I've configured a test trap but when the trap executes the service isn't updated. To troubleshoot, I've created a static passive check command that writes to the nagios.cmd pipe. When this event is ran on the CLI the service is updated. I then copied the same static check command into the event handler along with a command to write the trap details to a test file. When the trap is received the flat file is updated but the service still doesn't update.

This is the command that works from CLI but not as an event handler. The shell script is just copy/paste from the nagios passive service check directions.
/usr/local/nagios/libexec/submit_passive_service.sh "G407-7032" LostPowerEvent 2 "Device has lost power"

My first thought was file permissions but snmptrapd runs as root and snmptt is a member of the nagcmd group.

Thank you in advance for your assistance.

[scottwilkerson]

Can you share your submit_passive_service.sh as well as the command definition you have specified for submit_passive_service.sh

[gavnor]

This is the SNMPTT Definition that calls the command. I've tried multiple variations of the command including echoing directly into the pipe.

Code: Select all

### AUTO-GENERATED BY NXTI. DO NOT HAND-EDIT THIS FILE. ###
EVENT LostPowerEvent .1.3.6.1.4.1.22420.1.1.0.1 "Power Failure" Critical
FORMAT Received trap "$N" with variables "$+*"
EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N"  --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --bindings="$+*"
EXEC /usr/local/bin/snmptraphandling.py "$aR" "SNMP Traps" "$s" "$@" "" "SNMP Trap Received at $@ with variables $+*"
EXEC echo "$A - $aA - $c - $Fz - $N - $R - $r - $s" >> /usr/local/nagiosxi/var/NXTI_Write_Test
EXEC /usr/local/nagios/libexec/submit_passive_service.sh "G407-7032" LostPowerEvent 2 "Device has lost power"
SDESC
Lost Power Event
EDESC

submit_passive_checks.sh

Code: Select all

#!/bin/sh

# SUBMIT_CHECK_RESULT
# Written by Ethan Galstad ([email protected])
# Last Modified: 02-18-2002
#
# This script will write a command to the Nagios command
# file to cause Nagios to process a passive service check
# result.  Note: This script is intended to be run on the
# same host that is running Nagios.  If you want to
# submit passive check results from a remote machine, look
# at using the nsca addon.
#
# Arguments:
#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = return_code (An integer that determines the state
#       of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
#       3=UNKNOWN).
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service check)
#

echocmd="/bin/echo"

CommandFile="/usr/local/nagios/var/rw/nagios.cmd"

# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`

# create the command line to add to the command file
cmdline="[$datetime] PROCESS_SERVICE_CHECK_RESULT;$1;$2;$3;$4"

# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`%

[scottwilkerson]

You shouldn't be appending the the pipe, just write to it and it looks like some extra char
I would replace this

Code: Select all

`$echocmd $cmdline >> $CommandFile`%

With this

Code: Select all

/bin/printf "$cmdline\n" > $CommandFile

Also, are the other EXEC lines all running that are listed in the file?

Do you see an error in the nagios.log around the PROCESS_SERVICE_CHECK_RESULT external command?

[gavnor]

I can confirm that this exec is running

Code: Select all

EXEC echo "$A - $aA - $c - $Fz - $N - $R - $r - $s" >> /usr/local/nagiosxi/var/NXTI_Write_Test

I'm not sure about the others but the Trap service isn't updating so I would assume not.

I made the recommended changes to the submit script and manually ran it from CLI. The logs produced the following results.

Code: Select all

[1557251161] SERVICE NOTIFICATION: nagiosadmin;G407-7032;LostPowerEvent;CRITICAL;xi_service_notification_handler;Device has lost power
[1557251161] SERVICE ALERT: G407-7032;LostPowerEvent;CRITICAL;HARD;1;Device has lost power

I then triggered the trap but nothing was recorded in the nagios.log file. However, the test message was written to the NXTI_Write_Test file above.

[scottwilkerson]

I then triggered the trap but nothing was recorded in the nagios.log

Did you reset it back to OK before triggering the trap? If not it would make sense that it wasn't logged

[gavnor]

I did but the only thing logged was the host down state from the active host check.

[scottwilkerson]

Can you show the permissions here

Code: Select all

ls -l /usr/local/nagios/var/rw/nagios.cmd

[gavnor]

Code: Select all

prw-rw---- 1 nagios nagcmd 0 May  7 13:25 /usr/local/nagios/var/rw/nagios.cmd

I've also just turned on logging for external commands, passive service checks and event handlers.

When run from command line.

Code: Select all

[1557253622] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;G407-7032;LostPowerEvent;2;Device has lost power
[1557253622] PASSIVE SERVICE CHECK: G407-7032;LostPowerEvent;0;OK: No data received yet.
[1557253622] SERVICE NOTIFICATION: nagiosadmin;G407-7032;LostPowerEvent;CRITICAL;xi_service_notification_handler;Device has lost power
[1557253622] SERVICE ALERT: G407-7032;LostPowerEvent;CRITICAL;HARD;1;Device has lost power
[1557253622] GLOBAL SERVICE EVENT HANDLER: G407-7032;LostPowerEvent;CRITICAL;HARD;1;xi_service_event_handler

Nothing in the logs when the trap receiver handles it.

[gavnor]

What I also found interesting was that, in the admin UI SNMP Trap Interface, the Received Traps is always empty even though I've sent numerous traps, including the packaged "NXTI_Event" test trap. I know the traps are received and being matched because the event handlers that write to the test file are firing.

If I had to guess, I'd say this event handler processes the passive checks and populates the received traps so whatever is broken is affecting this, also.

Code: Select all

EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N"  --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --bindings="$+*"