SNMP Trap Problem

[WillemDH]

Hello,

I'm having issues with SNMP traps not being handled. I have some devices which can send test traps, but they do not arrive into the passive trap service I created. After looking in snmpttunknown.log and snmptt.log I did not find any traps received since 09/2013.
I've read several times through http://assets.nagios.com/downloads/nagi ... ios_XI.pdf, but could not find the answer untill I found a post on the forum saying something about /var/spool/snmptt, so I checked this directory and it seems I have 5801 traps waiting in the /var/spool/snmptt folder.
Also the snmpd service was stopped, I restarted it, but no test traps are coming in.

What could be the reason the traps are staying in the /var/spool/snmptt folder?

Please advice

Willem

Grtz

[sreinhardt]

This could be due to many things, so lets start with the basics.

Have you recently upgraded from snmptt 1.3 to 1.4?
What are the permissions on the spool directory and files:

Code: Select all

ls -lvd /var/spool/snmptt
ls -lv /var/spool/snmptt/ | tail -n 5

What user is your snmptt daemon running as?

Code: Select all

ps -ef | grep snmpt
grep -i 'daemon_uid' /etc/snmp/snmptt.ini

[WillemDH]

Hello,

No I didn't upgrade anything trap related.

Code: Select all

[root@xxx snmptt]# ls -lvd /var/spool/snmptt
drwxr-xr-x 2 nagios nagios 31772672 Jan 10 13:02 /var/spool/snmptt

Code: Select all

[root@xxx snmptt]# ls -lv /var/spool/snmptt/ | tail -n 5
-rw-r--r-- 1 root root 162 Jan 10 13:02 #snmptt-trap-1389355327243586
-rw-r--r-- 1 root root 162 Jan 10 13:02 #snmptt-trap-1389355328828235
-rw-r--r-- 1 root root 163 Jan 10 13:02 #snmptt-trap-1389355330298120
-rw-r--r-- 1 root root 163 Jan 10 13:02 #snmptt-trap-1389355331804272
-rw-r--r-- 1 root root 163 Jan 10 13:02 #snmptt-trap-1389355333345606

Code: Select all

[root@xxx snmptt]# ps -ef | grep snmpt
root      1891     1  0  2013 ?        00:01:14 /usr/bin/perl /usr/local/sbin/snmptt --daemon
root      1892  1891  0  2013 ?        00:00:00 /usr/bin/perl /usr/local/sbin/snmptt --daemon
root      2113  1892  0  2013 ?        00:00:00 python /usr/local/bin/snmptraphandling.py 10.54.18.131 SNMP Traps Normal 1378131728  An authenticationFailure trap signifies that the SNMP
root     11218     1  0 Jan09 ?        00:00:01 /usr/sbin/snmptrapd -Lsd -On -p /var/run/snmptrapd.pid
root     28988 32504  0 14:17 pts/0    00:00:00 grep snmpt

Code: Select all

[root@xxx snmptt]# grep -i 'daemon_uid' /etc/snmp/snmptt.ini
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt

Grtz

[lmiltchev]

Restart these two services:

Code: Select all

service snmptt restart
service snmptrapd restart

run the following commands, and show us the output:

Code: Select all

tail /var/log/messages
tail /var/log/snmptt/snmptt.log

[WillemDH]

[root@xxx ~]# service snmptt restart
Stopping snmptt: [FAILED]
Starting snmptt: [ OK ]
[root@xxx ~]# service snmptrapd restart
Stopping snmptrapd: [ OK ]
Starting snmptrapd: [ OK ]

Apparently snmptt was already stopped?

Jan 10 21:33:39 nagiosserver snmptrapd[26093]: 2014-01-10 21:33:39 NET-SNMP version 5.5 Stopped.
Jan 10 21:33:39 nagiosserver snmptrapd[26093]: Stopping snmptrapd
Jan 10 21:33:39 nagiosserver snmptrapd[28346]: NET-SNMP version 5.5
Jan 10 21:33:41 nagiosserver snmptt-sys[0]: SNMPTT v1.3 shutdown
Jan 10 21:33:41 nagiosserver snmptt-sys[0]: Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: SNMPTT v1.3 started
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: Loading /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: Finished loading 158 lines from /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: Loading /etc/snmp/snmptt.conf
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: Finished loading 1145 lines from /etc/snmp/snmptt.conf
Jan 10 21:33:45 nagiosserver snmptt-sys[0]: Could not convert user id 'snmptt' to a numeric UID

Just notice I do get some traps in now. I'll test Monday if my test traps arrive. Please overview my above logs. Is "Could not convert user id 'snmptt' to a numeric UID" normal?

Willem

[slansing]

We'll be able to take a bit more of an in depth look when we all get into work today, please let us know how things look. Is there a snmptt user on the system?:

Code: Select all

cat /etc/passwd | grep snmptt

[WillemDH]

We do not have an snmptt user:

Code: Select all

[root@nagios /]# cat /etc/passwd | grep snmptt
[root@nagios /]#

The test trap I can send with our DX8700 does arrive at the moment. So it appears to be working at the moment. I'll configure our DX60 this week. Maybe I'll better monitor the snmptrapd smptt and snmpd services so I know when they stop.

[sreinhardt]

OK, so we have a few issues, that I can see at this point that are combining to cause your issues.

1) /var/spool/snmptt is owned by nagios:nagios which at a bare minimum needs RW-RW-R--
2) the files within snmptt spool dir are owned by root, as snmptrapd is creating them that way, making them presently unreadable by snmptt daemon.
3) snmptt daemon is running as a user that does not exist on your system snmptt, likely causing the daemon not to start properly.

As for fixes, one immediate fix you can do is to switch the snmptt daemon to use root by altering the snmptt.ini to have daemon_uid set to nothing.

snmptt.ini

Code: Select all

daemon_uid = 

A more proper fix, would be to update to snmptt v1.4 if you are not there, create the snmptt user, and add it to the needed groups, chown the spool and log directories to snmptt:snmptt, chmod them to 664, keep the snmptt.ini daemon as snmptt, restart all snmp related services and see if things start populating.

I am happy to walk through either route you would like, if you want to go the latter, let's start with verifying what version of snmptt you presently have:

Code: Select all

rpm -qa | grep snmptt

[WillemDH]

Well as I'm using traps for only 4 or 5 devices at the moment, I think upgrading is the way to go, as we are currently on 1.3-3. I will need to send more traps in the future and looking at the changelog via http://www.snmptt.org/ it seems a good idea to upgrade.

Code: Select all

[root@nagios ~]# rpm -qa | grep snmptt
snmptt-1.3-3.nagios.noarch

[slansing]

Sounds good, this should cover what you need to do to upgrade:

http://snmptt.sourceforge.net/docs/snmp ... #Upgrading

http://sourceforge.net/projects/snmptt/files/snmptt/

Sreinhardt will also likely have some more input to offer on this.