Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

linkUp and linkDown traps coming through 6 times

https://support.nagios.com/forum/viewtopic.php?t=33981

Page 1 of 2

linkUp and linkDown traps coming through 6 times

Posted: Wed Jul 29, 2015 3:41 pm
by rkymtnhigh
Testing SNMP traps and when I trigger a linkDown, it shows in the snmptt.log file 6 times, and Nagios gets 6 traps. I get 6 alerts. Example:

Code: Select all

Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
And in Nagios:

Code: Select all

2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9936, COID: 370, CNID: 17685, CNMID: 17685
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9935, COID: 370, CNID: 17684, CNMID: 17684
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9934, COID: 370, CNID: 17683, CNMID: 17683
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9933, COID: 370, CNID: 17682, CNMID: 17682
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9932, COID: 370, CNID: 17681, CNMID: 17681
2015-07-29 14:22:04	Market Core Switch - Cisco	SNMP Traps	Service Problem	No	CRITICAL	team-ops	Nagios XI	A linkDown trap signifies that the SNMP entity, acting in 22067 Loopback10 softwareLoopback administratively down / ifIndex.22067 (INTEGER32):22067 ifDescr.22067 (OCTETSTR):Loopback10 ifType.22067 (INTEGER):softwareLoopback enterprises.9.2.2.1.1.20.22067	NID: 9931, COID: 370, CNID: 17680, CNMID:
Sometimes, when bringing the link back online, the linkUp gets triggered inconsistently, sometimes just once, sometimes twice, sometimes 6 times.

I feel like this might be a screwed up snmptt.conf file, I didn't entirely know what I was doing when I was importing them :)

I have gone thru the snmptt.conf file looking for duplicate linkUp and linkDown definitions, and found some, removed them, but still get the multiple entries when I trigger that trap.

Anyways, I'm open to suggestions, thanks in advance! Will provide more detail if needed.

RMH

Re: linkUp and linkDown traps coming through 6 times

Posted: Wed Jul 29, 2015 3:57 pm
by tgriep
After removing the duplicate entries, did you restart the snmptt daemon?

Code: Select all

service snmptt restart
Do you see multiple entries of them in the /var/log/snmp/snmptt.log file?

Re: linkUp and linkDown traps coming through 6 times

Posted: Wed Jul 29, 2015 4:08 pm
by rkymtnhigh
I did restart the snmptt service.

Yes, those traps show up 6 times each in the snmptt.log file.

Code: Select all

Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:52 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Wed Jul 29 14:29:53 2015 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.XX.X - A linkUp trap signifies that the SNMP entity, acting in an 22067 Loopback10 softwareLoopback up
Thank you!

Re: linkUp and linkDown traps coming through 6 times

Posted: Wed Jul 29, 2015 4:31 pm
by tgriep
Any chance you can generate a trap on the device to see if you get 6 traps on the XI system?

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 1:03 am
by Box293
In addition to what tgriep said.

What is the output of:

Code: Select all

ps aux | grep snmptt | grep -v grep
Stop the snmptt service:

Code: Select all

service snmptt stop
Force the trap to be sent.
The trap will be spooled into a file in /var/spool/snmptt/
Do you get one spooled file, or 6 spooled files all the same?

If you get one, then the device is only sending the trap once.
If you get six, then the device is sending the trap six times.

After this you can start the service again:

Code: Select all

service snmptt start

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 10:09 am
by rkymtnhigh
I generally get 6 traps in Nagios on linkDown, and sometimes 6, sometimes less on linkUp.

Output:

Code: Select all

# ps aux | grep snmptt | grep -v grep
root     30918  0.0  0.6 164828 12200 ?        Ss   Jul29   0:01 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt   30920  0.0  0.6 169124 13436 ?        Ss   Jul29   0:03 /usr/bin/perl /usr/sbin/snmptt --daemon
When I stop snmptt and force a trap, I get 12 entries in the /var/spool/snmptt directory. I force another trap and I get another 12.

Thanks guys,

RMH

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 11:13 am
by ssax
Please attach these files so that we can see what's going on:

Code: Select all

/etc/snmp/snmptt.conf
/etc/snmp/snmptt.ini
/etc/snmp/snmptrapd.conf

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 11:19 am
by rkymtnhigh
Attached. Thank you for the help.

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 4:02 pm
by ssax
I'm not seeing anything wrong with your config, I imported them into mine and they work just fine.

Run this command:

Code: Select all

tail -f /var/log/messages
Then send the test trap, if it shows more than once then it's definitely sending more than once.

Re: linkUp and linkDown traps coming through 6 times

Posted: Thu Jul 30, 2015 4:03 pm
by tmcdonald
Is it a possibility that the device is just falsely sending multiple traps for some reason? Let's run a tcpdump to confirm:

tcpdump port 162

Then force a trap and stop the running tcpdump. Post the output here. If there are many traps expected from multiple machines, you might limit the tcpdump to just the IP of the remote sending machine:

tcpdump port 162 and src host 192.168.1.100

of course replacing the IP with the correct one.
All times are UTC-05:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited