IPv6 and SNMP: Traps arrive, but not processed by snmptrapd

[gormank]

No, only from IPv4.

[rkennedy]

When you recompiled snmptrapd, what are the exact commands / parameters configured it with?

[gormank]

I'm not using the recompiled snmptrapd, or anything else. I'm using the one built with the system as part of the original installation last year.
The one I compiled wasn't installed--no make install, so it resides in my /tmp dir and only runs manually. It behaves exactly the same as the original snmptrapd.

./configure --with-mib-modules="mibII/ipv6" --enable-ipv6 --with-sys-contact=root --with-sys-location="Unknown" --with-logfile=/var/log/snmpd.log --with-persistent-directory=/var/net-snmp --with-default-snmp-version=2

[gormank]

So, to sort of start over ans summarize, IPv4 traps arrive and are working fine. v4 traps are handled by eth1. They are always logged.
IPv6 traps arrive on eth3.

# !tcpd
tcpdump -i eth3 -s 0 port 162
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
23:41:13.548781 IP6 2001:4888:a03:311f:c0:a:0:190.32880 > txslm2mlnag001v6.snmptrap: C=sp1der Trap(167) E:232 0.0.0.0 enterpriseSpecific s=11003 11180415 system.sysName.0="TXSLM2MCHP7004-OA" E:232.11.2.11.1=1 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: TXSLM2MCHP7004"

snmptrapd is configured as shown and listening. It has also been configured to listen on ::, the address of eth3, just udp6:162, etc. w/ the same result.

# ps -ef | grep snmptrapd | grep -v grep
root 10644 1 0 05:13 ? 00:00:01 /usr/sbin/snmptrapd -a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162,udp6:[::1]:162

# !netst
netstat -an | grep ":162 "
udp 0 0 0.0.0.0:162 0.0.0.0:*
udp 0 0 ::1:162 :::*

Firewalls are stopped so they aren't blocking, but behavior is the same when they're running.

# service iptables status; service ip6tables status
iptables: Firewall is not running.
ip6tables: Firewall is not running.

Since snmptt receives info from snmptrapd it has nothing to do with this issue. It has been tested by stopping snmptt and sending test traps regardless.

v6 traps sent from the nagios server to the nagios server v6 address are also not processed by snmptrapd.

# snmptrap -v 2c -c sp1der udp6:[2001:4888:a03:311f:c0:a:0:1f5]:162 42 linkUp
# snmptrap -v 2c -c sp1der udp6:[::1]:162 42 linkUp
# snmptrap -v 2c -c sp1der udp6:[::]:162 42 linkUp

Interestingly, I don't see the trap received in tcpdump.

# !tcpdu
tcpdump -i eth3 -s 0 port 162
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes

Maybe snmptrap isn't checking the address format/protocol and sending it over the right NIC. My bet is this isn't a valid test.

[Box293]

I've done a test on a fresh build of CentOS 6.7 x64.

I can make it work with this line in /etc/init.d/snmptrapd:

Code: Select all

OPTIONS="-Lsd udp:162 udp6:162 -p /var/run/snmptrapd.pid"

Here are the full steps that show how reproduce it:

Code: Select all

yum install -y net-snmp net-snmp-utils net-snmp-perl epel-release bzip2 tcpdump
yum install -y snmptt 
cd /tmp
wget http://assets.nagios.com/downloads/support/snmptrap-bins.tar.bz2
tar xjf snmptrap-bins.tar.bz2
cd snmptrap-bins/
cp ./* /usr/local/bin/
echo "disableAuthorization yes" > /etc/snmp/snmptrapd.conf
echo "traphandle default /usr/sbin/snmptthandler" >> /etc/snmp/snmptrapd.conf
service iptables stop
service ip6tables stop

Edit the file:
/etc/init.d/snmptrapd
Change this line:

Code: Select all

OPTIONS="-Lsd -p /var/run/snmptrapd.pid"

To this:

Code: Select all

OPTIONS="-Lsd udp:162 udp6:162 -p /var/run/snmptrapd.pid"

Then

Code: Select all

service snmptrapd stop
service snmptrapd start
service snmptt stop

In one terminal session run this:

Code: Select all

tcpdump -i lo -s 0 udp port 162

In another terminal session run these two commands:

Code: Select all

snmptrap -v 2c -c public ::1 '' NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 123456
snmptrap -v 2c -c public 127.0.0.1 '' NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 123456

Then you should find two spooled traps in /var/spool/snmptt/:

Code: Select all

cat /var/spool/snmptt/*

1460608056
localhost
UDP: [127.0.0.1]:35567->[127.0.0.1]
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:18:26.05
SNMPv2-MIB::snmpTrapOID.0 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification
NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate 123456


1460608067
UDP/IPv6: [::1]:45418
UDP/IPv6: [::1]:45418
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:18:36.69
SNMPv2-MIB::snmpTrapOID.0 NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification
NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatRate 123456

You'll also see this in the tcpdump:

Code: Select all

14:27:36.734505 IP localhost.35567 > localhost.snmptrap:  V2Trap(78)  system.sysUpTime.0=110605 S:1.1.4.1.0=E:8072.2.3.0.1 E:8072.2.3.2.1=123456

14:27:47.376019 IP6 localhost.45418 > localhost.snmptrap:  V2Trap(78)  system.sysUpTime.0=111669 S:1.1.4.1.0=E:8072.2.3.0.1 E:8072.2.3.2.1=123456

Also for reference:

Code: Select all

snmptrapd -v
NET-SNMP Version:  5.5

snmptrap -V
NET-SNMP version: 5.5

In addition to this I also tested on CentOS 7-1503. It's actually slightly different.
You need to edit the file: /etc/sysconfig/snmptrapd
And add the line:

Code: Select all

OPTIONS="udp:162 udp6:162"

Then:

Code: Select all

systemctl daemon-reload
systemctl restart snmptrapd.service

When I tried editing the file /usr/lib/systemd/system/snmptrapd.service with this:

Code: Select all

Environment=OPTIONS="-Lsd udp:162 udp6:162"

Only IPv4 traps were received.

[gormank]

Thanks for putting in the effort!
My system is RHEL 6.6 but I expect 6.7 is the same. Things change in 7.x...
From the init script, options is overwritten if the file exists so editing the init script has no effect.

Code: Select all

OPTIONS="-Lsd -p /var/run/snmptrapd.pid"
if [ -e /etc/sysconfig/snmptrapd ]; then
  . /etc/sysconfig/snmptrapd
fi

I changed the comma to a space, which mimics what you did.

# ps -ef | grep snmp
root 16993 1 0 21:42 ? 00:00:00 /usr/sbin/snmptrapd -a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162 udp6:[::1]:162

After a restart of snmptrapd and snmptt, sent a test trap. No change. No trap logged or processed.
BTW, I can send the trap w/ the traphandler python script, bypassing both snmptrapd and snmptt and get the alert...

No

[Box293]

From the init script, options is overwritten if the file exists so editing the init script has no effect.

I suggest commenting out everything in /etc/sysconfig/snmptrapd so they don't overwrite the options.

Then make the changes I suggested.

Try and keep things as simple as possible.

I changed the comma to a space, which mimics what you did.

I'm not sure what you're referencing here.

[gormank]

After a second look you have udp6:162. Updated /etc/sysconfig/snmptrapd

# grep -v ^# /etc/sysconfig/snmptrapd
OPTIONS="-a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162 udp6:162"

# ps -ef | grep snmp
root 6830 1 0 22:08 ? 00:00:00 /usr/sbin/snmptrapd -a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162 udp6:162

Woohoo!

2016-04-14 22:09:03 0.0.0.0(via UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190]:32880) TRAP, SNMP v1, community sp1der
CPQHOST-MIB::compaq Enterprise Specific Trap (CPQHOST-MIB::cpqHo2GenericTrap) Uptime: 2 days, 5:31:13.48
SNMPv2-MIB::sysName.0 = STRING: TXSLM2MCHP7004-OA CPQHOST-MIB::cpqHoTrapFlags = INTEGER: 1 CPQHOST-MIB::cpqHoGenericData = STRING: "HP Onboard Administrator Test Trap sent from enclosure: TXSLM2MCHP7004"

Days of hair pulling and resolved by removing a comma... I've tried many forms including udp6:162, but always had the comma.

# grep UDP/IPv6 /var/log/snmptt/snmptt.log
Thu Apr 14 22:09:03 2016 .1.3.6.1.4.1.232.0.11003 MAJOR "Status Events" UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190]:32880 - Generic trap (11003): HP Onboard Administrator Test Trap sent from enclosure: TXSLM2MCHP7004

But the trap doesn't result in a change on the host's SNMP traps monitor. At least I can move on to another problem.
Its not in the /var/log/snmptt/snmpttunknown.log

[gormank]

Note the difference in the lines below.

OPTIONS="-a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162,udp6:162"
OPTIONS="-a -A -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:162 udp6:162"

Interesting. Its in unconfigured objects as "UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190]:32880"

I'll use the wizard and see what it creates. Apparently it doesn't like my host w/ the address 2001:4888:a03:311f:c0:a:0:190.

Yep, it created a host named: UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190

Code: Select all

# grep -v ^# UDP_IPv6\:\ \[2001\:4888\:a03\:311f\:c0\:a\:0\:190.cfg

define host {
        host_name                       UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190
        use                             xiwizard_passive_host
        address                         UDP/IPv6: [2001:4888:a03:311f:c0:a:0:190
        max_check_attempts              5
        check_interval                  5
        retry_interval                  1
        check_period                    xi_timeperiod_24x7
        contacts                        nagiosadmin
        notification_interval           60
        notification_period             xi_timeperiod_24x7
        stalking_options                n
        icon_image                      passiveobject.png
        statusmap_image                 passiveobject.png
        _xiwizard                       passiveobject
        register                        1
        }

My guess is the trap handle python script can't handle the address in the trap.

[Box293]

Woohoo!

Excellent.

Interesting. Its in unconfigured objects as "UDP/IPv6: [2001:4888:a03c00:190]:32880"

I'll use the wizard and see what it creates. Apparently it doesn't like my host w/ the address 2001:4888:a03c00:190.

Yes, one of our techs picked this up about 20 minutes ago.

The problem is in the spooled trap, it is logged as:
UDP/IPv6: [2001:4888:a03c00:190]:32880
Instead of
2001:4888:a03c00:190

I'm going to spend some time looking into this today to see if I can get to the bottom of it.