SNMP Traps

[Envera IT]

I'm having a hell of a time getting SNMP Traps working in Nagios XI.

I've imported the MIB and made sure snmptt is translating the traps correctly. I've modified the status from Normal to Critical within snmptt.conf. At this point I have traps flowing into NagiosXI, but they show up in /var/log/messages instead of /var/log/snmptt/snmptt.log (also the documentation points to /var/log/snmptt.log instead of /var/log/snmptt/snmptt.log which is where my snmptt.log file exists. Also the device isn't showing up in unconfigured objects within the XI GUI.

Looking for advice on what to look into at this point.

[tgriep]

Did you go through this guide to enable XI to receive SNMP Traps?
https://assets.nagios.com/downloads/nag ... os_XI.pdf
If not, run through it and see if this resolves the issue for you.

If you still have problems, can you post the files in this folder so we can review them?

Code: Select all

/etc/snmp

Can you post an example of the trap that is in the messages file as well?

[gormank]

Snmptrapd is the trap receiver, and it logs in messages. Do you see these entries?
Do you see snmptt entries in messages, or just the snmptt.log?
Are there files in /var/spool/snmptt/?

My system has
/var/log/snmptt/snmptt.log
/var/log/snmptt/snmpttsystem.log
/var/log/snmptt/snmpttunknown.log

What's in those files, assuming they exist?

What are their timestamps? ll /var/log/snmptt/*log

[ssax]

In addition to tgriep's and gormank's posts, by default snmptrapd will put the traps into /var/log/messages so that's why you are seeing them there.

Sounds like you have a good idea of how it works but I'll include the info below just in case it answers some questions.

This is the general flow of how SNMP traps work:

Device -> XI Server -> snmptrapd -> snmptt -> Nagios XI

Here's how it works in greater detail:

1. The device sends a SNMP trap with say an OID of .1.3.6.1.6.3.1.1.5.1 to the Nagios XI server.

2. The snmptrapd service receives the trap and then runs the default handler for traps (in this case SNMPTT)
- Taken from /etc/snmp/snmptrapd.conf

Code: Select all

traphandle default /usr/sbin/snmptthandler

3. SNMPTT reads the trap and does some processing on it based on it's configuration (translate IP of sender into DNS name, strip domain, all configurable in /etc/snmp/snmptt.ini).

4. SNMPTT doesn't know anything about the traps in your MIB files, the MIB files on the system are just used for translation from .1.3.6.1.6.3.1.1.5.1 into coldStart. You need to process the MIB file that contains your traps to get them into the /etc/snmp/snmptt.conf file which SNMPTT reads to match against to see if it should do anything with it (.1.3.6.1.6.3.1.1.5.1).

5. Since you've run addmib on the MIB file containing your traps (in this case /usr/share/snmp/mibs/SNMPv2-MIB.txt) it processes the trap and puts it into a format SNMPTT understands and changes the EXEC line (see below) to execute the snmptraphandling.py script (that's what puts it into Nagios).

Code: Select all

EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
FORMAT A coldStart trap signifies that the SNMP entity, $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A coldStart trap signifies that the SNMP entity, $*"
SDESC
A coldStart trap signifies that the SNMP entity,
supporting a notification originator application, is
reinitializing itself and that its configuration may
have been altered.
Variables:
EDESC

So SNMPTT says "Hey, I received a trap with OID .1.3.6.1.6.3.1.1.5.1, do I know anything about it? Let me check my /etc/snmp/snmptt.conf file. Oh, I see it matches the coldStart event (from above), I will run this EXEC line now (which happens to put it into Nagios)."

You can read more about SNMPTT and what those lines mean (and how you can change them if you want) here:

http://snmptt.sourceforge.net/docs/snmptt.shtml

[Envera IT]

Did you go through this guide to enable XI to receive SNMP Traps?
https://assets.nagios.com/downloads/nag ... os_XI.pdf
If not, run through it and see if this resolves the issue for you.

If you still have problems, can you post the files in this folder so we can review them?

Code: Select all

/etc/snmp

Can you post an example of the trap that is in the messages file as well?

I did run through the document but had issues getting it going. I looked through some other documentation online which got me abit further along, but still not all the way. Totally possible that I have something miss-configured at this point.

Trap example (basically a switched PDU sends a trap whenever it trips a power outlet).

Code: Select all

Jun 21 12:55:05 localhost snmptrapd[1308]: 2016-06-21 12:55:05 192.168.242.50(via UDP: [192.168.242.50]:161->[10.0.1.161]) TRAP, SNMP v1, community redacted#012#011.1.3.6.1.4.1.1418.4 Enterprise Specific Trap (5) Uptime: 0:11:29.52#012#011.1.3.6.1.4.1.1418.4.3.1.2.2 = STRING: "Outlet3"#011.1.3.6.1.4.1.1418.4.3.1.3.2 = INTEGER: 2

[Envera IT]

Snmptrapd is the trap receiver, and it logs in messages. Do you see these entries?
Do you see snmptt entries in messages, or just the snmptt.log?
Are there files in /var/spool/snmptt/?

My system has
/var/log/snmptt/snmptt.log
/var/log/snmptt/snmpttsystem.log
/var/log/snmptt/snmpttunknown.log

What's in those files, assuming they exist?

What are their timestamps? ll /var/log/snmptt/*log

I see the snmptrapd entries in messages but I don't see any snmptt entries in messages other than when I restarted snmptt earlier, that does show up.


I took a long break from this issue, so not much activity recently on this. The Jun 21 event in snmpttsystem.log is from me rebooting snmptt, nothing else in the log.


-rw-r--r-- 1 root root 0 Feb 18 03:10 snmptt.debug
-rw-r--r-- 1 root root 3.2K Feb 17 21:15 snmptt.debug-20160218
-rw-r--r-- 1 root root 0 Feb 18 10:59 snmptt.log
-rw-rw-r-- 1 snmptt snmptt 407 Jun 21 21:06 snmpttsystem.log
-rw-rw-r-- 1 snmptt snmptt 485 Mar 12 2015 snmpttsystem.log-20150313
-rw-rw-r-- 1 snmptt snmptt 2.1K Feb 17 21:15 snmpttsystem.log-20160218
-rw-rw-r-- 1 snmptt snmptt 1.9K Feb 18 11:07 snmpttsystem.log-20160221
-rw-rw-r-- 1 snmptt snmptt 407 Apr 7 14:36 snmpttsystem.log-20160408
-rw-rw-r-- 1 snmptt root 0 Feb 21 03:22 snmpttunknown.log
-rw-rw-r-- 1 snmptt root 782K Feb 18 11:07 snmpttunknown.log-20160221

[Envera IT]

In addition to tgriep's and gormank's posts, by default snmptrapd will put the traps into /var/log/messages so that's why you are seeing them there.

Sounds like you have a good idea of how it works but I'll include the info below just in case it answers some questions.

This is the general flow of how SNMP traps work:

Device -> XI Server -> snmptrapd -> snmptt -> Nagios XI

Here's how it works in greater detail:

1. The device sends a SNMP trap with say an OID of .1.3.6.1.6.3.1.1.5.1 to the Nagios XI server.

2. The snmptrapd service receives the trap and then runs the default handler for traps (in this case SNMPTT)
- Taken from /etc/snmp/snmptrapd.conf

Code: Select all

traphandle default /usr/sbin/snmptthandler

3. SNMPTT reads the trap and does some processing on it based on it's configuration (translate IP of sender into DNS name, strip domain, all configurable in /etc/snmp/snmptt.ini).

4. SNMPTT doesn't know anything about the traps in your MIB files, the MIB files on the system are just used for translation from .1.3.6.1.6.3.1.1.5.1 into coldStart. You need to process the MIB file that contains your traps to get them into the /etc/snmp/snmptt.conf file which SNMPTT reads to match against to see if it should do anything with it (.1.3.6.1.6.3.1.1.5.1).

5. Since you've run addmib on the MIB file containing your traps (in this case /usr/share/snmp/mibs/SNMPv2-MIB.txt) it processes the trap and puts it into a format SNMPTT understands and changes the EXEC line (see below) to execute the snmptraphandling.py script (that's what puts it into Nagios).

Code: Select all

EVENT coldStart .1.3.6.1.6.3.1.1.5.1 "Status Events" Normal
FORMAT A coldStart trap signifies that the SNMP entity, $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A coldStart trap signifies that the SNMP entity, $*"
SDESC
A coldStart trap signifies that the SNMP entity,
supporting a notification originator application, is
reinitializing itself and that its configuration may
have been altered.
Variables:
EDESC

So SNMPTT says "Hey, I received a trap with OID .1.3.6.1.6.3.1.1.5.1, do I know anything about it? Let me check my /etc/snmp/snmptt.conf file. Oh, I see it matches the coldStart event (from above), I will run this EXEC line now (which happens to put it into Nagios)."

You can read more about SNMPTT and what those lines mean (and how you can change them if you want) here:

http://snmptt.sourceforge.net/docs/snmptt.shtml

This is very helpful, I will look over my configs tomorrow. I feel like I went through many of these steps when I was working on this a few months ago, but at this point I just need to start troubleshooting from the beginning again.

[Box293]

Let us know how you go.

This tutorial goes into detail as to how the traps actually work from when they enter the server through to the Nagios services.
https://support.nagios.com/kb/article.php?id=77

It's long, but it's worth setting up the scenario as a way of playing with SNMP as I find that's the best way to learn.

[Envera IT]

Well I spun up a fresh Nagios XI vm and got traps working on that in 5 minutes flat. But still having issues with the production setup. I'll be comparing the configs between the two instances today to see whats off and will report back again.

[tgriep]

I think I found the issue. The settings in the snmptrapd.conf file are incorrect to they are not getting passed from the snmptrapd daemon to the snmptt daemon correctly.
Can you edit the /etc/snmp/snmptrapd.conf file
Delete the following line

Code: Select all

traphandle default /usr/local/bin/snmptraphandling.py

and uncomment the following line
#traphandle default /usr/sbin/snmptthandler

Save the file and restart the snmptrapd daemon by running the following

Code: Select all

service snmptrapd restart

Try that and see if this works for you.