SNMP Traps not processed into alerts, but visible in NXTI

[911comp]

Nagios XI running older version without NXTI was processing SNMP Traps defined in "snmptt.conf"and forwarding them into alerts.
After Nagios XI got was upgraded to 5.5.10, everything in "snmptt.conf" got commented out, saying that NXTI handles all the traps now.

Problem is that all the trap definitions got migrated into NXTI, but EXEC action is missing from all of them, so now i can see received traps in NXTI, but they are not getting passed to nagios "SNMP Traps" service.
snmptt.conf sample:

Code: Select all

# This trap definition is being managed by NXTI
#EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Warning
#FORMAT Device reinitialized (coldStart)
#EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Device reinitialized (coldStart) $*"
#SDESC
#A coldStart trap signifies that the SNMPv2 entity, acting
#in an agent role, is reinitializing itself and that its
#configuration may have been altered.
#EDESC
##

snmptt.conf.nxti sample:

Code: Select all

EVENT nsNotifyRestart .1.3.6.1.4.1.8072.4.0.3 "Status Events" Normal
FORMAT Received trap "$N" with variables "$+*"
EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N"  --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --bindings="$+*"
SDESC
An indication that the agent has been restarted.
This does not imply anything about whether the configuration has
changed or not (unlike the standard coldStart or warmStart traps)
Variables:

EDESC

snmptt.debug sample:

Code: Select all

OID of received trap: .1.3.6.1.4.1.9.9.41.2.0.1.  Will attempt to translate to text
    OID found in cache:  '.1.3.6.1.4.1.9.9.41.2.0.1' -> 'clogMessageGenerated'
  Translated to clogMessageGenerated
EXEC command:php /usr/local/nagiosxi/scripts/nxti.php --event_name="clogMessageGenerated"  --event_oid=".1.3.6.1.4.1.9.9.41.2.0.1" --numeric_oid=".1.3.6.1.4.1.9.9.41
.2.0.1" --symbolic_oid="clogMessageGenerated" --community="" --trap_hostname="xxxxxxxx" --trap_ip="xxxxxxxxx" --agent_hostname="xxxxxxxxx" --agent_ip
="xxxxxxxxx" --category="Status Events" --severity="Normal" --uptime="337:19:03:08.03" --datetime="2019-03-01 14:27:36" --unixtime="1551468456" --bindings="clog
HistFacility.161072:PLATFORM_ENV clogHistSeverity.161072:2 clogHistMsgName.161072:FRU_PS_ACCESS clogHistMsgText.161072:FRU Power Supply is not responding clogHistTim
estamp.161072:337:19:03:08.00"
Sleeping for 5 seconds

Attached is screenshot from NXTI interface, with missing exec

[swolf]

Hi @911comp,

That looks like a pretty serious oversight on our part. Sorry about any inconvenience that's caused you. We have some changes for the Manage MIBs/NXTI pages planned for XI 5.6, and I'll make sure that this particular issue is handled in that fix.

In the meantime, here's a workaround that can get you to where you were previously, without NXTI:

1) In /etc/snmp/snmptt.ini, go to the end of the file. In the snmptt_conf_file directive, remove the line that says "snmptt.conf.nxti" and save the file. Note that this will cause snmptt to ignore anything that's set up by NXTI.

2) Move snmptt.conf to snmptt.conf.bak (so that you still have access to these old configs), and create a new snmptt.conf with the same permissions as the old file.

3) Do one of the following:

a) In Admin->System Extensions -> Manage MIBs, click the "Process All Traps" button in the upper right.

-or-

b) Run the following bash code in the terminal

Code: Select all

for x in /usr/share/snmp/mibs/*; do
  if [ ! -d $x ]; then
    addmib $x
  fi
done

Either of these should do the SNMP Trap processing as it occurred prior to NXTI's release. It won't re-do any hand modifications that you've made to the snmptt.conf file, but it will fix the issues you're having with monitoring at the moment.

[911comp]

Thank you for prompt response. Known SNMP Traps are now getting passed over to Nagios XI service.


I am also seeing weird behavior with unknown_traps.
snmptt is configured to pass unknown_traps to nagios with Warning status(see snmptt.ini below).

As you can see from debug below, 1st unknown trap from host is getting processed as intended: snmptraphandling.py is called with parameters according to format specified, but after second trap is received for same host then snmptraphandling.py is getting called with parameters from old trap PLUS parameters from a new trap. So new trap values never get passed to nagios.
I have no idea what is causing this behavior. Wondering if you can help with this?


snmptt.ini [EXEC] block

Code: Select all

[Exec]

# Set to 1 to allow EXEC statements to execute.  Should normally be left on unless you
# want to temporarily disable all EXEC commands
exec_enable = 1

# Set to 1 to allow PREEXEC statements to execute.  Should normally be left on unless you
# want to temporarily disable all PREEXEC commands
pre_exec_enable = 1

# If defined, the following command will be executed for ALL unknown traps.  Passed to the
# command will be all standard and enterprise variables, similar to unknown_trap_log_file
# but without the newlines.
unknown_trap_exec = /usr/local/bin/snmptraphandling.py
#unknown_trap_exec = /usr/local/bin/snmptraphandling_sed.sh
# FORMAT line that is passed to the unknown_trap_exec command.  If not defined, it
# defaults to what is described in the unknown_trap_exec setting.  The following
# would be *similar* to the default described in the unknown_trap_exec setting
# (all on one line):
# $x !! $X: Unknown trap ($o) received from $A at: Value 0: $A Value 1: $aR
# Value 2: $T Value 3: $o Value 4: $aA Value 5: $C Value 6: $e Ent Values: $+*
unknown_trap_exec_format = $r" "SNMP Traps" "Warning" "$@" "" "Unknown trap ($o).Values:$+*

#unknown_trap_exec_format =$r" "SNMP Traps" "Warning" "$@" "" "Unknown trap ($o).Values: $*

# Set to 1 to escape wildards (* and ?) in EXEC, PREEXEC and the unknown_trap_exec
# commands.  Enable this to prevent the shell from expanding the wildcard
# characters.  The default is 1.
exec_escape = 1

snmptt.debug sample

Code: Select all

Sleeping for 5 seconds

Processing file: #snmptt-trap-1551482289303926
Agent IP address was blank, so setting to the same as the host IP address of **Server_IP**

Agent IP address (**Server_IP**) is the same as the host IP, so copying the host name: **Server_Hostname**

Trap received from **Server_Hostname**: SNMPv2-SMI::enterprises.40540.2.0.1.0.7.0.7
Exact match of trap NOT found in EVENT hash table

Looking for wildcards in the EVENT hash table


Trap not defined...



Unknown trap EXEC line:
Unknown trap EXEC command:/usr/local/bin/snmptraphandling.py "**Server_Hostname**" "SNMP Traps" "Warning" "1551482289" "" "Unknown trap (.1.3.6.1.4.1.40540.2.0.1.0.7.0.7).Values:stor2rrdSendTrap.0.7.0.7:111"
Sleeping for 5 seconds

Processing file: #snmptt-trap-1551482292747804
Agent IP address was blank, so setting to the same as the host IP address of **Server_IP**

Agent IP address (**Server_IP**) is the same as the host IP, so copying the host name: **Server_Hostname**

Trap received from **Server_Hostname**: SNMPv2-SMI::enterprises.40540.2.0.1.0.7.0.7
Exact match of trap NOT found in EVENT hash table

Looking for wildcards in the EVENT hash table


Trap not defined...



Unknown trap EXEC line:
Unknown trap EXEC command:/usr/local/bin/snmptraphandling.py "**Server_Hostname**" "SNMP Traps" "Warning" "1551482289" "" "Unknown trap (.1.3.6.1.4.1.40540.2.0.1.0.7.0.7).Values:stor2rrdSendTrap.0.7.0.7:111" "**Server_Hostname**" "SNMP Traps" "Warning" "1551482292" "" "Unknown trap (.1.3.6.1.4.1.40540.2.0.1.0.7.0.7).Values:stor2rrdSendTrap.0.7.0.7:222"
Sleeping for 5 seconds

[swolf]

I also haven't ever seen anything like this. If you change the entry for unknown_trap_exec_format to something completely different (for instance, "testing $r testing $+* testing", do you continue to see the appending behavior?

Similarly, if you send a third trap with the same OID, does the log show details for all three getting passed in?

I'm also seeing that the quotes are structured oddly in unknown_trap_exec_format. Are you intentionally quoting the spaces in order to pass the empty string? I doubt it's the cause of the issue but it might be worth looking into if the change above 'fixes' things.

If this turns out to be an actual bug, it'd be best to head over to the sourceforge page (https://sourceforge.net/p/snmptt/bugs/s ... osed-fixed) and open a new ticket there. The maintainer doesn't get much traffic, but on the last ticket that was opened he responded within a day.

[911comp]

thanks for suggestions @swolf.

Similarly, if you send a third trap with the same OID, does the log show details for all three getting passed in?

yeah, it keeps appending new new trap parameters to command. i got it up to 7 till i gave up. snmptt restart clears that up.

Ill play around with formats and parameters. thanks again.

[swolf]

No problem!

I'm going to lock this thread, but feel free to start a new one if you make any headway or have any other concerns.