SNMP Trap Problem

[sreinhardt]

Slight alteration to the doc, and slansings post. We have made the commands in that doc into a script. You should just need to run the following, then restart the snmptrapd and snmptt services.

Code: Select all

cd /tmp
wget http://assets.nagios.com/downloads/nagiosxi/scripts/NagiosXI-SNMPTrap-setup.sh
chmod +x ./NagiosXI-SNMPTrap-setup.sh
sh ./NagiosXI-SNMPTrap-setup.sh

[WillemDH]

Nice. Thanks for the script. I've downloaded it and executed it as asked. Checking version again gives:

Code: Select all

[root@nagios tmp]# rpm -qa | grep snmptt
snmptt-1.4-0.6.beta2.el6.noarch

Is it correct the version is still in beta?

Now I send some new test traps, but they do not seem to arrive, while yesterday I received a trap from this device.

[WillemDH]

Apparently the script also restarts the snmptt and snmptrapd services. This is the output:

Code: Select all

Wed Jan 15 08:50:13 2014 SNMPTT v1.3 shutdown
Wed Jan 15 08:50:13 2014 Total traps received=49,Total traps translated=9,Total traps ignored=0,Total unknown traps=42
Wed Jan 15 08:50:17 2014 SNMPTT v1.4beta2 started
Wed Jan 15 08:50:17 2014 Loading /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Wed Jan 15 08:50:17 2014 Finished loading 158 lines from /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Wed Jan 15 08:50:17 2014 Loading /etc/snmp/snmptt.conf
Wed Jan 15 08:50:17 2014 Finished loading 1145 lines from /etc/snmp/snmptt.conf
Wed Jan 15 08:50:17 2014 Changing to UID: snmptt (497)
Wed Jan 15 09:12:18 2014 SNMPTT v1.4beta2 started
Wed Jan 15 09:12:18 2014 Loading /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Wed Jan 15 09:12:18 2014 Finished loading 158 lines from /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Wed Jan 15 09:12:18 2014 Loading /etc/snmp/snmptt.conf
Wed Jan 15 09:12:18 2014 Finished loading 1145 lines from /etc/snmp/snmptt.conf
Wed Jan 15 09:12:18 2014 Changing to UID: snmptt (497)

I cannot find the test trap in /var/log/snmptt/snmpttunknown.log and /var/log/snmptt.log seems to stay empty.

I executed the previous commands again:

Code: Select all

ls -lvd /var/spool/snmptt
drwxr-xr-x 2 snmptt snmptt 31772672 Oct 30  2012 /var/spool/snmptt

Code: Select all

ls -lv /var/spool/snmptt/ | tail -n 5
total 0

Code: Select all

 ps -ef | grep snmpt
root      2113     1  0  2013 ?        00:00:00 python /usr/local/bin/snmptraphandling.py 10.10.3.3 SNMP Traps Normal 1378131728  An authenticationFailure trap signifies that the SNMP
root     16459     1  0 09:12 ?        00:00:00 /usr/sbin/snmptrapd -Lsd -On -p /var/run/snmptrapd.pid
root     16586     1  0 09:12 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt   16587 16586  0 09:12 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
root     27664 22595  0 09:22 pts/0    00:00:00 grep snmpt

Code: Select all

grep -i 'daemon_uid' /etc/snmp/snmptt.ini
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt

Code: Select all

service snmptt restart
service snmptrapd restart

gives tail /var/log/messages

Code: Select all

Jan 15 09:23:55 nagios snmptt-sys[16587]: SNMPTT v1.4beta2 shutdown
Jan 15 09:23:55 nagios snmptt-sys[16587]: Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Jan 15 09:23:59 nagios snmptt-sys[29909]: SNMPTT v1.4beta2 started
Jan 15 09:23:59 nagios snmptt-sys[29909]: Loading /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 15 09:23:59 nagios snmptt-sys[29909]: Finished loading 158 lines from /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 15 09:23:59 nagios snmptt-sys[29909]: Loading /etc/snmp/snmptt.conf
Jan 15 09:23:59 nagios snmptt-sys[29909]: Finished loading 1145 lines from /etc/snmp/snmptt.conf
Jan 15 09:23:59 nagios snmptt-sys[29912]: Changing to UID: snmptt (497)

and tail /var/log/snmptt/snmptt.log still empty

Code: Select all

cat /etc/passwd | grep snmptt
snmptt:x:497:496:SNMP Trap Translator:/var/spool/snmptt:/sbin/nologin

Grtz

[sreinhardt]

Thanks for running those other commands too, that narrowed down some things. Looks like most items are as they should be, let's verify permissions on the actual log files and that snmptt is in the nagios and nagcmd groups.

Code: Select all

grep 'snmptt' /etc/group
ll -d /var/log/snmptt/
ll /var/log/snmptt/

Also I noted that you had said the file "/var/log/snmptt.log" when it should be /var/log/snmptt/snmptt.log, if you're using vim, it might not complain that this is a folder not a file. Otherwise your processes are running great, the permissions on the spool directory looks great, and not having anything in the spool directory means it should be getting picked up correctly.

Also, yep for some reason it's been listed in beta for a year plus with no major changes. There are however some issues with 1.3 that warrant updating even though it is listed as beta.

[WillemDH]

First of all, I meant /var/log/snmptt/snmptt.log (typo)

The commands:

Code: Select all

grep 'snmptt' /etc/group
nagios:x:500:nagios,apache,snmptt
nagcmd:x:501:nagios,apache,snmptt
snmptt:x:496:

Code: Select all

ll -d /var/log/snmptt/
drwxr-xr-x 2 snmptt snmptt 4096 Jan 16 03:09 /var/log/snmptt/

Code: Select all

 ll /var/log/snmptt/
total 1368
-rw-rw-r-- 1 root root      0 Jan 14 03:11 snmptt.log
-rw-rw-r-- 1 root root  36623 Sep  2 16:22 snmptt.log-20130908
-rw-rw-r-- 1 root root    139 Nov 14 10:57 snmptt.log-20131115
-rw-rw-r-- 1 root root 683479 Jan 10 21:25 snmptt.log-20140111
-rw-rw-r-- 1 root root   1524 Jan 13 13:01 snmptt.log-20140114
-rw-r--r-- 1 root root      0 Jan 16 03:09 snmpttsystem.log
-rw-r--r-- 1 root root    617 Aug 29 23:12 snmpttsystem.log-20130901
-rw-r--r-- 1 root root    452 Nov 14 10:57 snmpttsystem.log-20131115
-rw-r--r-- 1 root root   2920 Jan 10 21:33 snmpttsystem.log-20140111
-rw-r--r-- 1 root root   1467 Jan 15 09:23 snmpttsystem.log-20140116
-rw-rw-r-- 1 root root  21470 Jan 14 19:09 snmpttunknown.log
-rw-rw-r-- 1 root root  27745 Sep  1 02:09 snmpttunknown.log-20130901
-rw-rw-r-- 1 root root   6654 Sep  2 16:02 snmpttunknown.log-20130908
-rw-rw-r-- 1 root root 574309 Jan 10 21:25 snmpttunknown.log-20140111
-rw-rw-r-- 1 root root   2394 Jan 12 18:30 snmpttunknown.log-20140113

The entries in the snmpttunknown.log are from before the upgrade and come from a sanswitch that was miconfigured. I just treid to send another test trap, but nothing is coming in.

The last traps I got from the device I test with (DX8700) is 2014-01-13 13:01:55 An authenticationFailure trap signifies that the SNMP,
Then I also got some traps in the evening in the snmpttunknown.log

Code: Select all

Tue Jan 14 19:09:41 2014: Unknown trap (.1.3.6.1.4.1.231.2.54.2.0.2501) received from genspip1 at:
Value 0: genspip1
Value 1: 10.00.00.00
Value 2: 160:20:20:29.81
Value 3: .1.3.6.1.4.1.231.2.54.2.0.2501
Value 4: 10.54.18.248
Value 5: 53 00 65 00 72 00 76 00 65 00 72 00 56 00 69 00  65 00 77 00
Value 6: .1.3.6.1.4.1.231.2.54.2
Value 7:
Value 8:
Value 9:
Value 10:
Ent Value 0: .1.3.6.1.4.1.231.2.54.2.1.1=SAPASPR03
Ent Value 1: .1.3.6.1.4.1.231.2.54.2.1.2=1389722981
Ent Value 2: .1.3.6.1.4.1.231.2.54.2.1.3=allCPU
Ent Value 3: .1.3.6.1.4.1.231.2.54.2.1.4=35

An the 15the I upgraded and since then no traps anymore..

Restarted all the snmp services once more.. This is the log:

Code: Select all

Jan 16 17:14:07 nagios snmptt-sys[29736]: SNMPTT v1.4beta2 shutdown
Jan 16 17:14:07 nagios snmptt-sys[29736]: Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Jan 16 17:14:08 nagios snmptt-sys[30878]: SNMPTT v1.4beta2 started
Jan 16 17:14:08 nagios snmptt-sys[30878]: Loading /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 16 17:14:08 nagios snmptt-sys[30878]: Finished loading 158 lines from /usr/share/snmp/mibs/processed_mibs/FJDARY-E102.txt
Jan 16 17:14:08 nagios snmptt-sys[30878]: Loading /etc/snmp/snmptt.conf
Jan 16 17:14:08 nagios snmptt-sys[30878]: Finished loading 1145 lines from /etc/snmp/snmptt.conf
Jan 16 17:14:08 nagios snmptt-sys[30881]: Changing to UID: snmptt (497)
Jan 16 17:14:10 nagios snmptrapd[29178]: 2014-01-16 17:14:10 NET-SNMP version 5.5 Stopped.
Jan 16 17:14:10 nagios snmptrapd[29178]: Stopping snmptrapd
Jan 16 17:14:10 nagios snmptrapd[30911]: NET-SNMP version 5.5
Jan 16 17:14:16 nagios snmpd[29013]: Received TERM or STOP signal...  shutting down...
Jan 16 17:14:16 nagios snmpd[31027]: NET-SNMP version 5.5

I have no idea where it's going wrong. Seems like nothing even comes in while I'm 100 % sure traps are sent to Nagios..

[sreinhardt]

Biggest thing I am noticing, those files are root:root with no write for anyone else. Let's try altering the ownership and restarting snmptt again. This might be something we need to patch in that script for people upgrading.

Code: Select all

chown -R snmptt:snmptt /var/log/snmptt/*
service snmptt restart.

The reason the snmptt-system.log is still updating for restart times and such, is that it is written by the daemon before dropping privileges.

[WillemDH]

Executed

Code: Select all

chown -R snmptt:snmptt /var/log/snmptt/*
service snmptt restart.

And sent test trap, but nothing received. Please advice.

Willem

[sreinhardt]

Can I have you put in a ticket please, and we will setup a remote. At this point, so much seems exactly right, that I think I just need to take a look at the system and see what is going wrong. In the meantime, if you would enable debugging in snmptt.ini and enable the debug log files at the same location. Restart the snmptt daemon and we can take a peek at those too. Otherwise we can do that in the remote too, no worries either way.

[lmiltchev]

@WillemDH

You opened a new support ticket in our email ticketing system. We will continue communicating via emails. I am locking this topic.