SNMP Traps not working anymore

[lavignp]

Hi, I seem to be having the same problem with my SNMP traps not working anymore on my Nagios XI system. It was working last year, but the last time they seemed to work was last August. I have upgraded twice since then, and just performed the most recent update to give it one more try. I am currently on 2012R2.9.

I have tried to follow all of the steps in the "Integrating SNMP Traps With Nagios XI" guide to get it working again, but the traps just seem to evaporate. I have tried following the advice on numerous forum posts, but I nothing seems to be getting to the heart of it.

I have tried upgrading the latest version of net-snmp, as well as the latest version of snmptt (upgraded using yum). I have also tried re-installing the MIB's in /usr/share/snmp/mibs/

If I tail my logs, I can see the traps coming in:

[root@v-nagios01 snmptt]# tail -f /var/log/messages
Feb 14 12:01:35 v-nagios01 snmptrapd[43736]: 2014-02-14 12:01:35 10.160.100.100(via UDP: [10.160.100.00]:63373->[10.160.100.200]) TRAP, SNMP v1, community Foo #012#011.1.3.6.1.4.1.311.1.1.3.1.2 Cold Start Trap (0) Uptime: 0:00:00.00#012

I set up the SNMP Trap for the host, and I also looked in the Unconfigured Objects. Nothing shows up.

I have looked in /var/log/snmptt/snmpttunknown.log and snmptt.log, and both are empty. I have tried to send the test trap as indicated in the guide, and nothing shows up.

I have tried stopping a restarting services, and nothing fixes it. However, whenever I start up snmptt I get the following:

[root@v-nagios01 tmp]# service snmptt start
Starting snmptt: Unlinked OID in HOST-RESOURCES-MIB: host ::= { mib-2 25 }
Undefined identifier: mib-2 near line 9 of /usr/share/snmp/mibs/hostmib.mib
Did not find 'hrMIBAdminInfo' in module HOST-RESOURCES-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt)
Did not find 'hrStorage' in module HOST-RESOURCES-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt)
Did not find 'hrDevice' in module HOST-RESOURCES-MIB (/usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt)
Unlinked OID in HOST-RESOURCES-TYPES: hostResourcesTypesModule ::= { hrMIBAdminInfo 4 }
Undefined identifier: hrMIBAdminInfo near line 7 of /usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt
Unlinked OID in HOST-RESOURCES-TYPES: hrFSTypes ::= { hrDevice 9 }
Undefined identifier: hrDevice near line 226 of /usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt
Unlinked OID in HOST-RESOURCES-TYPES: hrDeviceTypes ::= { hrDevice 1 }
Undefined identifier: hrDevice near line 108 of /usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt
Unlinked OID in HOST-RESOURCES-TYPES: hrStorageTypes ::= { hrStorage 1 }
Undefined identifier: hrStorage near line 37 of /usr/share/snmp/mibs/HOST-RESOURCES-TYPES.txt
Cannot adopt OID in HOST-RESOURCES-TYPES: hostResourcesTypesModule ::= { hrMIBAdminInfo 4 }

Please help.

[sreinhardt]

The unlinked oids are likely causing some issues, although they likely would not cause the daemon to fail startup. Basically they are saying that you have a mib that requires another mib for it to contain the full oid string. As for the issues of snmptraps not working, we can check a few things right off the bat.

Code: Select all

grep -i 'snmp\|addmib' /usr/local/bin/
grep -i 'snmp\|addmib' /usr/local/sbin/
grep -i 'snmp\|addmib' /usr/sbin/
cat /etc/snmp/snmptrapd.conf

grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
grep -i -m 5 'exec' /etc/snmp/snmptt.conf
grep -i 'nag' /etc/group
grep -i 'snmp' /etc/group

ll /var/log/snmptt/
ll -d /var/log/snmptt/
ll /var/spool/snmptt
ll -d /var/spool/snmptt

If you could respond with the results of those commands please, it would get us off to a great start.

[lavignp]

Okay, here goes

[root@monitor01 ~]# grep -i 'snmp\|addmib' /usr/local/bin/
[root@monitor01 ~]# grep -i 'snmp\|addmib' /usr/local/bin/
[root@monitor01 ~]# grep -i 'snmp\|addmib' /usr/local/sbin/
[root@monitor01 ~]# grep -i 'snmp\|addmib' /usr/sbin/
[root@monitor01 ~]# cat /etc/snmp/snmptrapd.conf
traphandle default /usr/local/sbin/snmptthandler
disableAuthorization yes

[root@monitor01 ~]# grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt
[root@monitor01 ~]# grep -i -m 5 'exec' /etc/snmp/snmptt.conf
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The newRoot trap indicates that the sending agent has $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A top ologyChange trap is sent by a bridge when any of $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Notification that the trigger indicated by the object $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Notification that the rising threshold was met for triggers $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Notification that the falling threshold was met for triggers $*"
[root@monitor01 ~]# grep -i 'nag' /etc/group
nagios500:nagios,apache,snmptt
nagcmd501:nagios,apache,snmptt
[root@monitor01 ~]# grep -i 'snmp' /etc/group
nagios500:nagios,apache,snmptt
nagcmd501:nagios,apache,snmptt
snmptt496:

[root@monitor01 ~]# ll /var/log/snmptt/
total 4772
-rw-rw-r-- 1 root root 0 Sep 29 01:35 snmptt.log
-rw-rw-r-- 1 root root 110497 Sep 8 00:47 snmptt.log-20130908
-rw-rw-r-- 1 root root 250853 Sep 15 00:47 snmptt.log-20130915
-rw-rw-r-- 1 root root 50494 Sep 22 00:47 snmptt.log-20130922
-rw-rw-r-- 1 root root 106321 Sep 26 07:47 snmptt.log-20130929
-rw-r--r-- 1 root root 0 Feb 16 03:23 snmpttsystem.log
-rw-r--r-- 1 root root 621 Sep 24 13:13 snmpttsystem.log-20130925
-rw-r--r-- 1 root root 779 Sep 26 07:53 snmpttsystem.log-20130929
-rw-r--r-- 1 root root 1712 Feb 13 17:55 snmpttsystem.log-20140214
-rw-r--r-- 1 root root 2387 Feb 14 10:01 snmpttsystem.log-20140216
-rw-rw-r-- 1 root root 0 Sep 29 01:35 snmpttunknown.log
-rw-rw-r-- 1 root root 1180446 Sep 8 01:15 snmpttunknown.log-20130908
-rw-rw-r-- 1 root root 1218812 Sep 15 01:25 snmpttunknown.log-20130915
-rw-rw-r-- 1 root root 1182748 Sep 22 01:10 snmpttunknown.log-20130922
-rw-rw-r-- 1 root root 721299 Sep 26 07:50 snmpttunknown.log-20130929
[root@monitor01 ~]# ll -d /var/log/snmptt/
drwxr-xr-x 2 snmptt snmptt 4096 Feb 16 03:23 /var/log/snmptt/
[root@monitor01 ~]# ll /var/spool/snmptt
total 0

Thank you

[sreinhardt]

My mistake, these:

Code: Select all

grep -i 'snmp\|addmib' /usr/local/bin/
grep -i 'snmp\|addmib' /usr/local/sbin/
grep -i 'snmp\|addmib' /usr/sbin/

Should actually be:

Code: Select all

ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
ls -lva /usr/sbin | grep -i 'snmp\|addmib'

Otherwise I see two other commands I seem to have missed:

Code: Select all

ll -d /var/spool/snmptt
yum list installed | grep -i snmp

And some issues right off the bat, mainly that all of your log files are owned by root:root and your daemon is running as snmptt. You should be able to correct them right away with:

Code: Select all

chown root:nagios /etc/snmp/snmptt.ini /etc/snmp/snmptt.conf /etc/snmp /usr/local/bin/addmib
chmod g+w /etc/snmp/snmptt.ini /etc/snmp
chmod g+x /usr/local/bin/addmib

chown -R snmptt:snmptt /var/spool/snmptt /var/log/snmptt
chmod -R ug+wx /var/spool/snmptt /var/log/snmptt

[lavignp]

I was wondering what you were asking for, but I figured you knew something I didn't...

Hehe

Okay, here are the updated results:

Code: Select all

[root@monitor01 ~]# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root root      804 Feb 14 09:10 addmib
-r-xr-xr-x   1 root root     4817 Feb  3 12:13 snmpkey
-rwxr-xr-x   1 root root     2078 Feb 14 09:10 snmptraphandling.py
-rwxr-xr-x   1 root root    30438 Feb 14 09:10 snmpttconvertmib
[root@monitor01 ~]# ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
[root@monitor01 ~]# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root root       30744 Nov 22 07:01 snmpd
-rwxr-xr-x   1 root root       30776 Nov 22 07:01 snmptrapd
-rwxr-xr-x   1 root root      177466 Oct 22  2012 snmptt
-rwxr-xr-x   1 root root        6493 Oct 22  2012 snmptthandler

And the missed ones:

Code: Select all

[root@monitor01 ~]# ll -d /var/spool/snmptt
drwxr-xr-x 2 snmptt snmptt 4096 Jan 18 10:44 /var/spool/snmptt
[root@monitor01 ~]# yum list installed | grep -i snmp
net-snmp.x86_64                       1:5.5-49.el6                     @base
net-snmp-devel.x86_64                 1:5.5-49.el6                     @base
net-snmp-libs.x86_64                  1:5.5-49.el6                     @base
net-snmp-perl.x86_64                  1:5.5-49.el6                     @base
net-snmp-utils.x86_64                 1:5.5-49.el6                     @base
perl-Net-SNMP.noarch                  5.2.0-4.el6                      @epel
perl-SNMP_Session.noarch              1.12-4.el6                       @base
php-snmp.x86_64                       5.3.3-27.el6_5                   @updates
snmptt.noarch                         1.4-0.9.beta2.el6                @epel

I have updated the permissions as indicated, and will run some tests now.

thanks!

[sreinhardt]

Cool, let me know how it works out for you!

[lavignp]

No luck with the permissions changes. I stopped and restarted the three services, and I still cannot seem to get the traps to come in.

[sreinhardt]

OK, I'd like to verify permissions post changes, then if you could send me the /etc/snmp/{snmptt.ini,snmptt.conf, snmptrapd.conf} files please. Feel free to PM if you would like, although there really shouldn't be anything sensitive in there. Most likely we will need to enable debugging and take a look at those logs, but let's start with the basics.

Code: Select all

ll /var/log/snmptt/
ll -d /var/log/snmptt/
ll /var/spool/snmptt
ll -d /var/spool/snmptt

[lavignp]

Here are the perms:

Code: Select all

[root@monitor01 ~]# ll /var/log/snmptt/
total 4776
-rwxrwxr-- 1 snmptt snmptt       0 Sep 29 01:35 snmptt.log
-rwxrwxr-- 1 snmptt snmptt  110497 Sep  8 00:47 snmptt.log-20130908
-rwxrwxr-- 1 snmptt snmptt  250853 Sep 15 00:47 snmptt.log-20130915
-rwxrwxr-- 1 snmptt snmptt   50494 Sep 22 00:47 snmptt.log-20130922
-rwxrwxr-- 1 snmptt snmptt  106321 Sep 26 07:47 snmptt.log-20130929
-rwxrwxr-- 1 snmptt snmptt     596 Feb 18 13:54 snmpttsystem.log
-rwxrwxr-- 1 snmptt snmptt     621 Sep 24 13:13 snmpttsystem.log-20130925
-rwxrwxr-- 1 snmptt snmptt     779 Sep 26 07:53 snmpttsystem.log-20130929
-rwxrwxr-- 1 snmptt snmptt    1712 Feb 13 17:55 snmpttsystem.log-20140214
-rwxrwxr-- 1 snmptt snmptt    2387 Feb 14 10:01 snmpttsystem.log-20140216
-rwxrwxr-- 1 snmptt snmptt       0 Sep 29 01:35 snmpttunknown.log
-rwxrwxr-- 1 snmptt snmptt 1180446 Sep  8 01:15 snmpttunknown.log-20130908
-rwxrwxr-- 1 snmptt snmptt 1218812 Sep 15 01:25 snmpttunknown.log-20130915
-rwxrwxr-- 1 snmptt snmptt 1182748 Sep 22 01:10 snmpttunknown.log-20130922
-rwxrwxr-- 1 snmptt snmptt  721299 Sep 26 07:50 snmpttunknown.log-20130929
[root@monitor01 ~]# ll -d /var/log/snmptt/
drwxrwxr-x 2 snmptt snmptt 4096 Feb 16 03:23 /var/log/snmptt/
[root@monitor01 ~]# ll /var/spool/snmptt
total 0
[root@monitor01 ~]# ll -d /var/spool/snmptt
drwxrwxr-x 2 snmptt snmptt 4096 Jan 18 10:44 /var/spool/snmptt

I attached the files as requested. i had to zip the snmptt.conf file, because it was 8mb, and doesn't want to upload.

[sreinhardt]

I think your only real issue is the change of location for snmptraphandler. Try this and restart snmptt and snmptrapd

/etc/snmp/snmptrapd:

Code: Select all

traphandle default /usr/local/sbin/snmptthandler
TO:
traphandle default /usr/sbin/snmptthandler