Traps Not Processed on 2014R1.2

[mikew]

I have run the NagiosXI-SNMPTrap-setup.sh to update the snmptt.

Traps are not being processed as seen from this command:
ls /var/spool/snmptt|wc -l
61

This seems to be the major problem as the traps arrive at the /var/spool/snmptt but never get processed.


ll /var/spool
drwxrwxr-x 2 snmptt snmptt 4096 Jan 18 11:44 snmptt

I also changed these permissions to nagios:nagios as the new script makes snmptt a member of the nagios group.

ll /var/spool
drwxrwxr-x 2 nagios nagios 4096 Jan 18 11:44 snmptt

I do not understand why the script has changed snmptrapd.conf to this, which for sure will break all updates to 2014 if people have snmptrapd configured:

# Example configuration file for snmptrapd
# No traps are handled by default, you must edit this file!
# authCommunity log,execute,net public
# traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script cold

My current snmptrapd.conf does not reflect that of the script, here is what is in place:
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler

I am currently receiving traps as seen here, there are no unknown traps:
/var/log/messsages
Jun 29 11:52:12 localhost snmptt[13389]: .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.5.10 - A linkUp trap signifies that the SNMP entity, acting in an 49 up up

tail snmptt.log
Fri Jun 27 05:14:37 2014 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.5.10 - A linkUp trap signifies that the SNMP entity, acting in an 49 up up
Fri Jun 27 05:14:37 2014 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.5.10 - A linkUp trap signifies that the SNMP entity, acting in an 49 up up
Fri Jun 27 05:14:37 2014 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.5.10 - A linkUp trap signifies that the SNMP entity, acting in an 49 up up

snmpttsystem.log
Sun Jun 29 10:22:38 2014 SNMPTT v1.4beta2 started
Sun Jun 29 10:22:38 2014 Loading /etc/snmp/snmptt.conf
Sun Jun 29 10:22:38 2014 Finished loading 709 lines from /etc/snmp/snmptt.conf
Sun Jun 29 10:22:38 2014 Changing to UID: snmptt (501)

ps aux|grep snmptt
root 13388 0.0 0.1 147596 9592 ? Ss 11:52 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt 13389 0.0 0.1 149824 10476 ? Ss 11:52 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon

ps aux|grep snmptrapd
root 10781 0.0 0.0 190920 2340 ? Ss 10:21 0:00 /usr/sbin/snmptrapd -Lsd -On -p /var/run/snmptrapd.pid

[sreinhardt]

Most things look perfectly fine, although I would agree that it seems strange that the script would set:

# Example configuration file for snmptrapd
# No traps are handled by default, you must edit this file!
# authCommunity log,execute,net public
# traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script cold

opposed to, what it should be:

disableAuthorization yes
traphandle default /usr/sbin/snmptthandler

I see only two pieces of information missing that would point us in the right direction, so let's try the following:

Code: Select all

ls -lart /var/spool/snmptt | head 25
grep -i -B 5 -A 5 'A linkUp trap signifies that' /etc/snmp/snmptt.conf

By default, that trap would never actually get passed to core\xi since it does not have a proper exec line, but it is entirely possible that you changed this already.

[mikew]

ls -lart /var/spool/snmptt
total 376
drwxr-xr-x. 9 root root 4096 May 16 04:18 ..
-rw-rw-r-- 1 nagios nagios 219 Jun 27 05:14 #snmptt-trap-1403867677146568
-rw-rw-r-- 1 nagios nagios 221 Jun 27 05:14 #snmptt-trap-1403867677301192
-rw-rw-r-- 1 nagios nagios 221 Jun 27 05:14 #snmptt-trap-1403867677455455
-rw-rw-r-- 1 nagios nagios 349 Jun 27 05:14 #snmptt-trap-1403867677609638
-rw-rw-r-- 1 nagios nagios 219 Jun 27 05:14 #snmptt-trap-1403867679778090
-rw-rw-r-- 1 nagios nagios 334 Jun 27 05:14 #snmptt-trap-1403867684267870
-rw-rw-r-- 1 nagios nagios 219 Jun 27 06:17 #snmptt-trap-1403871463008742
-rw-rw-r-- 1 nagios nagios 334 Jun 27 06:17 #snmptt-trap-1403871467500154
-rw-rw-r-- 1 nagios nagios 221 Jun 27 07:59 #snmptt-trap-1403877574051256
-rw-rw-r-- 1 nagios nagios 219 Jun 27 07:59 #snmptt-trap-1403877576006320
-rw-rw-r-- 1 nagios nagios 334 Jun 27 07:59 #snmptt-trap-1403877580490353
-rw-rw-r-- 1 nagios nagios 221 Jun 27 08:04 #snmptt-trap-1403877874043915
-rw-rw-r-- 1 nagios nagios 219 Jun 27 08:04 #snmptt-trap-1403877876043253
-rw-rw-r-- 1 nagios nagios 350 Jun 27 08:04 #snmptt-trap-1403877880044271
-rw-rw-r-- 1 nagios nagios 220 Jun 27 08:27 #snmptt-trap-1403879230796744
-rw-rw-r-- 1 nagios nagios 220 Jun 27 08:36 #snmptt-trap-1403879803326187
-rw-rw-r-- 1 nagios nagios 220 Jun 27 12:59 #snmptt-trap-1403895558777037
-rw-rw-r-- 1 nagios nagios 218 Jun 27 12:59 #snmptt-trap-1403895561589182
-rw-rw-r-- 1 nagios nagios 333 Jun 27 12:59 #snmptt-trap-1403895566079888




grep -i -B 5 -A 5 'A linkUp trap signifies that' /etc/snmp/snmptt.conf
EDESC
#
#
#
EVENT linkUp .1.3.6.1.6.3.1.1.5.4 "Status Events" Normal
FORMAT A linkUp trap signifies that the SNMP entity, acting in an $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkUp trap signifies that the SNMP entity, acting in an $*"
SDESC
A linkUp trap signifies that the SNMP entity, acting in an
agent role, has detected that the ifOperStatus object for
one of its communication links left the down state and
transitioned into some other state (but not into the
notPresent state). This other state is indicated by the
included value of ifOperStatus.

[sreinhardt]

Oh sure, go ahead and change the default exec line on me. So that part looks great, but it worries me that you have spooled files from the 27th still, leading me to believe that the snmptt user cannot remove them. Try adding W permissions for group to /var/spool/snmptt/ and remove the existing files. Then let's turn debugging on for snmptt and give snmptt a restart just to be 100% sure that it picks up the past changes to the exec line and the new debug changes.

I'm sure you know these commands, but for others in the future:

Code: Select all

chmod g+w /var/spool/snmptt
rm -rf /var/spool/snmptt/*

modify /etc/snmp/snmptt.ini to:
DEBUGGING = 2
#DEBUGGING_FILE =
DEBUGGING_FILE = /var/log/snmptt/snmptt.debug
#DEBUGGING_FILE_HANDLER =
DEBUGGING_FILE_HANDLER = /var/log/snmptt/snmptthandler.debug

service snmptt restart

While we are in there, let's check the nagios.cmd file with:

Code: Select all

ls -lart /usr/local/nagios/var/rw/

Send a few snmp traps in again, and if you could tar up those debug files and send them over please.

[mikew]

/usr/local/nagios/var/rw/
-rw-rw-r-- 1 nagios nagcmd 1845 Jun 28 09:23 nsca.dump
srw-rw---- 1 nagios nagcmd 0 Jun 30 11:14 nagios.qh
drwxrwsr-x. 2 nagios nagcmd 4096 Jun 30 11:14 .
prw-rw---- 1 nagios nagcmd 0 Jun 30 11:29 nagios.cmd
drwxrwxr-x. 6 nagios nagios 4096 Jun 30 11:30 ..


Here are the traps stuck in snmptt

1404149466
<UNKNOWN>
UDP: [192.168.5.10]:161->[192.168.5.5]
.1.3.6.1.2.1.1.3.0 24:3:22:20.18
.1.3.6.1.6.3.1.1.4.1.0 .1.3.6.1.6.3.1.1.5.4
.1.3.6.1.2.1.2.2.1.1.56 56
.1.3.6.1.2.1.2.2.1.7.56 up
.1.3.6.1.2.1.2.2.1.8.56 up

1404149471
<UNKNOWN>
UDP: [192.168.5.10]:161->[192.168.5.5]
.1.3.6.1.2.1.1.3.0 24:3:22:24.68
.1.3.6.1.6.3.1.1.4.1.0 .1.3.6.1.4.1.9.6.1.101.0.151
.1.3.6.1.4.1.9.6.1.101.2.3.1.0 "%STP-W-PORTSTATUS: gi8: STP status Forwarding^M
"
.1.3.6.1.4.1.9.6.1.101.2.3.2.0 1
.1.3.6.1.4.1.9.6.1.101.57.2.8.1.0 56
.1.3.6.1.4.1.9.6.1.101.57.2.8.2.0 0


Logs

snmpthandler.tar.gz

snmptt.tar.gz

[sreinhardt]

The only two traps in the debug logs are the two you show in the post just above. Unfortunately we didn't verify if those were properly setup or not. Could you run a few of the traps for linkup like:

Fri Jun 27 05:14:37 2014 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.5.10 - A linkUp trap signifies that the SNMP entity, acting in an 49 up up

Then send a those logs again please?

[mikew]

So I have solved the problem. I had traps working fine with previous versions of 2012 and 2014 collecting traps for hosts in the Nagios interface as well as NSTI. When I installed 2014R1.2 it change permissions on files apparently and then, since I saw an update for snmptt was recommended I updated using the script NagiosXI-SNMPTrap-setup.sh The script changed the permissions of /var/spool/snmptt to nagios:nagios which meant that snmptt could not remove the incoming traps. The script also changed these group permissions to:

nagios500:nagios,apache,snmptt
nagcmd501:nagios,apache,snmptt

That is what broke everything with traps. I returned permissions to snmptt:snmptt for /var/spool/snmptt and everything is processed normally. You can close this.