linkUp and linkDown traps coming through 6 times

[rkymtnhigh]

Testing SNMP traps and when I trigger a linkDown, it shows in the snmptt.log file 6 times, and Nagios gets 6 traps. I get 6 alerts. Example:

Code: Select all

Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up

And in Nagios:

Code: Select all

2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9936, COID: 370, CNID: 17685, CNMID: 17685
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9935, COID: 370, CNID: 17684, CNMID: 17684
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9934, COID: 370, CNID: 17683, CNMID: 17683
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9933, COID: 370, CNID: 17682, CNMID: 17682
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9932, COID: 370, CNID: 17681, CNMID: 17681
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9931, COID: 370, CNID: 17680, CNMID: 

Sometimes, when bringing the link back online, the linkUp gets triggered inconsistently, sometimes just once, sometimes twice, sometimes 6 times.

I feel like this might be a screwed up snmptt.conf file, I didn't entirely know what I was doing when I was importing them

I have gone thru the snmptt.conf file looking for duplicate linkUp and linkDown definitions, and found some, removed them, but still get the multiple entries when I trigger that trap.

Anyways, I'm open to suggestions, thanks in advance! Will provide more detail if needed.

RMH

[tgriep]

After removing the duplicate entries, did you restart the snmptt daemon?

Code: Select all

service snmptt restart

Do you see multiple entries of them in the /var/log/snmp/snmptt.log file?

[rkymtnhigh]

I did restart the snmptt service.

Yes, those traps show up 6 times each in the snmptt.log file.

Code: Select all

Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up

Thank you!

[tgriep]

Any chance you can generate a trap on the device to see if you get 6 traps on the XI system?

[Box293]

In addition to what tgriep said.

What is the output of:

Code: Select all

ps aux | grep snmptt | grep -v grep

Stop the snmptt service:

Code: Select all

service snmptt stop

Force the trap to be sent.
The trap will be spooled into a file in /var/spool/snmptt/
Do you get one spooled file, or 6 spooled files all the same?

If you get one, then the device is only sending the trap once.
If you get six, then the device is sending the trap six times.

After this you can start the service again:

Code: Select all

service snmptt start

[rkymtnhigh]

I generally get 6 traps in Nagios on linkDown, and sometimes 6, sometimes less on linkUp.

Output:

Code: Select all

# ps aux | grep snmptt | grep -v grep
root     30918  0.0  0.6 164828 12200 ?        Ss   Jul29   0:01 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt   30920  0.0  0.6 169124 13436 ?        Ss   Jul29   0:03 /usr/bin/perl /usr/sbin/snmptt --daemon

When I stop snmptt and force a trap, I get 12 entries in the /var/spool/snmptt directory. I force another trap and I get another 12.

Thanks guys,

RMH

[ssax]

Please attach these files so that we can see what's going on:

Code: Select all

/etc/snmp/snmptt.conf
/etc/snmp/snmptt.ini
/etc/snmp/snmptrapd.conf

[rkymtnhigh]

Attached. Thank you for the help.

[ssax]

I'm not seeing anything wrong with your config, I imported them into mine and they work just fine.

Run this command:

Code: Select all

tail -f /var/log/messages

Then send the test trap, if it shows more than once then it's definitely sending more than once.

[tmcdonald]

Is it a possibility that the device is just falsely sending multiple traps for some reason? Let's run a tcpdump to confirm:

tcpdump port 162

Then force a trap and stop the running tcpdump. Post the output here. If there are many traps expected from multiple machines, you might limit the tcpdump to just the IP of the remote sending machine:

tcpdump port 162 and src host 192.168.1.100

of course replacing the IP with the correct one.