Send and alert on SNMP Traps to NagiosXI

[dfmco]

NSTI 3.02
NAGIOS XI 2014r1.2
I have attached my snmptt configurations here.

snmptt.conf

snmptt.ini

I Think I am REALLY close. My NSTI is getting traps but the traps are not getting into NagiosXI. I have loaded the MIBS but I am sure that the EXEC command is not working. Since this is my first time trying this, I want to make sure I am doing it correctly. This is the command I used to import the MIB into SNMPTT

Code: Select all

snmpttconvertmib --in=CISCO-TCP-MIB.my.txt --out=/etc/snmp/snmptt.conf --net_snmp_perl

My thought is that I also need to add a command to link this to Nagios. I have seen a few examples but many are different and I am not sure if that will damage the existing snmptt.conf file (should I delete and start a fresh one)? Would I append something similar code below to my snmpttconvertmib command? Do I keep the --net_snmp_perl? Since the path /usr/local/nagios/libexec/eventhandlers/ does not exist in NagiosXI and I can't find the submit_check_result file anywhere on the system, does anyone know the proper path and executable to use for a default NagiosXI install on CentOS6.5?

Code: Select all

snmpttconvertmib --in=CISCO-TCP-MIB.my.txt --out=/etc/snmp/snmptt.conf --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1' --net_snmp_perl

My second issue is that would I also have to do the same for all of the other stock MIBS to fix the EXEC command there as well like below?

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Normal
FORMAT Link down on interface $1.  Admin state: $2.  Operational state: $3
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1.  Admin state: $2.  Operational state: $3"

My guess is that qpage is not the correct command.

I would also like a CRITICAL alert and page to go out from NagiosXI when a few of these are received. My best guess is that this is done in the snmptt.conf file somehow but I am not sure how or what variable I would need to modify. Would that be the "Normal" keyword that follows "Status Events" as in the code above? If so, what are the options other than Normal?

Finally, for the stuff that I don't want to alert, is there a default catch-all at the end of the snmptt.conf that will forward these on to NagiosXI? I saw an example but like above the patch is wrong. I would like to make sure the other options are appropriate before trying.

Code: Select all

EVENT CatchAll .1.* "SNMP Traps" Critical
FORMAT $D
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result "$r"  
"snmp_traps" 2 "$O: $1 $2 $3 $4 $5"

[dfmco]

I was able to get a little bit further. I used the addmib command which populated my snmptt.conf file. MIBs have dependencies that you have to satisfy.

Code: Select all

addmib /usr/share/snmp/mibs/<MIB TO ADD>
service snmptt restart
service snmptrapd restart

I also figured out that I was correct on the Normal/Warning/Critical designations. I am going to test that tomorrow to make sure the alerts are coming into Nagios.

[sreinhardt]

First thing first, I have to throw it out there, technically we very much suggest against installing multiple nagios products on the same system. This is mostly for potential load issues since most of our systems will use quite a bit of resources on their own, and even more together. Can it be done, yes absolutely, but you need to keep in mind that it will potential cause load and IO issues, depending on your particular setup.

Now that that's out of the way, you absolutely need to use addmib, so that the proper exec lines will be generated. Submit_check_result, is not a standard XI or core plugin, and I'm honestly not sure where people are getting it from. You should be using /usr/local/bin/snmptraphandler.py (path might be a bit different), for all exec lines that you wish to send to a local XI system. Since you have used snmpttconvertmib already, I would suggest removing those entries from snmptt.conf, as snmptt will always pull the first match, and rerunning the imports with addmib as you have done. Once this is done, you will need to restart just the snmptt service. Send a few traps and give it a test.

Once we get items coming into XI properly, then we can work on state changes to warning or critical.

If you have not seen it before, even with your current install I would suggest running through the Integrating snmp traps with nagios XI doc here:
http://assets.nagios.com/downloads/nagi ... ith_XI.pdf

[dfmco]

I thought NSTI was a plug in that had to run on the same box. I was not aware that it needed a separate machine. Good information. I had assumed because all the docs I could find showed the install local on the Nagios box. Is there a FAQ that I can contribute to that may aid others?

So I moved my current config to config.old and re-ran addmib for the SNMP traps I want to collect. I am getting config traps now. I set the level from Normal to Warning and that seems to work as well. The only notifications I am getting are config management at this time.

I have Cisco EEM throwing a custom trap. How would I set that trap to show in the interface? That may be more of an EEM question.

How can I get a trap to alert? Would that be based on the Critical/Warning settings only or is there another step?

[sreinhardt]

You are right that there is integration between NSTI and XI, but it is not a plugin in the normal sense for XI. Could you point out where it was showing core\XI installations on the same system as NSTI? I would love to correct those docs and put in more clarification.

Glad that got your traps coming into XI properly! As for EEM, I would probably suggest contacting cisco about any possible mibs they might have to help with that. Otherwise if needed, we can work through snmptt debug output and try to get something together that will work well enough for it.

Traps alerting is another kind of tricky point. If you notice at the top of each OID section within a snmptt.conf file, you have an EVENT line. The word NORMAL, CRITICAL, or whatever may be in place is used as a static status for nagios results. If you know a trap should always be a particular state you can change it to NORMAL, WARNING, CRITICAL, SEVERE, etc. If you need a trap to have dynamic states it gets a bit more tricky, but I can walk you through it if you would like.

[dfmco]

OK, I see all configuration traps and I figured out the severity but I am having trouble seeing a few traps as they are going into the "unknown.log".

Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.166159=EIGRP-IPv4 2604: Neighbor 10.252.136.1 (Serial0/0/0:0) is down: Interface PEER-TERMINATION received
Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.166159=EIGRP-IPv4 2604: Neighbor 10.252.136.1 (Serial0/0/0:0) is down: Interface PEER-TERMINATION received
Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.136096=EIGRP-IPv4 2604: Neighbor 10.252.32.2 (Serial0/3/0:0) is down: interface down
Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.136097=Interface Serial0/3/0:0, changed state to administratively down
Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.29514=EIGRP-IPv4 2604: Neighbor 10.252.32.1 (Serial0/3/1:0) is down: holding time expired
Ent Value 3: .1.3.6.1.4.1.9.9.41.1.2.3.1.5.136098=Line protocol on Interface Serial0/3/0:0, changed state to down

For EIGRP I have this in my snmptt.conf file:

Code: Select all

EVENT EIGRP .1.3.6.1.4.1.9.9.41.1.2.3.1.5 "Status Events" Critical
FORMAT EIGRP ROUTE CHANGE, $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "EIGRP ROUTE CHANGE, $*"
SDESC
A EIGRP trap signifies that the routing table has changed.
Variables:
EDESC

And for Link Down, I have this:

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Critical
FORMAT LINK DOWN $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "LINK DOWN $*"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state).  This other state is indicated by the included value
of ifOperStatus.
Variables:
  1: ifIndex
  2: ifAdminStatus
  3: ifOperStatus
EDESC

Should I have an "*" at the end of the EIGRP entry? I am not sure if ".1.3.6.1.4.1.9.9.41.1.2.3.1.5" in my match string is matching the actual event of ".1.3.6.1.4.1.9.9.41.1.2.3.1.5.166159" the 166159 seems to change depending on the device, interface and alert. Am I reading that correctly?

For LinkDown, I see that my OID is not matching but I am confused as I have another install that works fine for link down using ".1.3.6.1.6.3.1.1.5.3". Am I reading the OID correctly or am I missing something basic?

[dflick]

Also, can polling time be changed in the snmp trap service so that a trap stays active for longer than the default 1 minute? I noticed that traps are being overwritten very quickly.

[sreinhardt]

Should I have an "*" at the end of the EIGRP entry? I am not sure if ".1.3.6.1.4.1.9.9.41.1.2.3.1.5" in my match string is matching the actual event of ".1.3.6.1.4.1.9.9.41.1.2.3.1.5.166159" the 166159 seems to change depending on the device, interface and alert. Am I reading that correctly?

This is exactly what I would suggest, for the exact reason that devices and often interfaces will append another identifier at the end of this particular oid. An asterisks will work as a catch-all within snmptt configs.

Also, can polling time be changed in the snmp trap service so that a trap stays active for longer than the default 1 minute? I noticed that traps are being overwritten very quickly.

Absolutely, I would suggest setting a longer freshness timeout in the xiwizard_snmptrap_service template. This should allow for everything else to act the same, and simply increase that trap reset to a reasonable amount if time.

[dflick]

OK, for the links down I am not seeing the alert but I do get this in the event log:
2014-07-10 05:49:12Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;10.252.0.32;SNMP Traps;0;LINK RECOVERED 8 Serial0/3/0:0 propPointToPointSerial Keepalive OK / ifIndex.8 (INTEGER32):8 ifDescr.8 (OCTETSTR):Serial0/3/0:0 ifType.8 (INTEGER):propPointToPointSeri
2014-07-10 05:49:07Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;10.252.0.32;SNMP Traps;2;LINK DOWN 8 Serial0/3/0:0 propPointToPointSerial administratively down / ifIndex.8 (INTEGER32):8 ifDescr.8 (OCTETSTR):Serial0/3/0:0 ifType.8 (INTEGER):propPointToPoint

Does this mean the trap fails to process? How can I found out what is failing to fix it?

[dflick]

OK, that may have been old info. I think it is working:
HELOTES-32-R1View service status details for this host SNMP TrapsPassive Only Check Critical 1s 1/1 2014-07-10 20:13:50 LINK DOWN 8 Serial0/3/0:0 propPointToPointSerial administratively down / ifIndex.8 (INTEGER32):8 ifDescr.8 (OCTETSTR):Serial0/3/0:0 ifType.8 (INTEGER):propPointToPointSerial enterprises.9.2.2.1.1.20.8 ():administratively dow
It looks like it is alerting as well! I will do some testing but we may be OK to close this thread by tomorrow!