SNMP traps processed only from one interface

[jsimek]

Hello Nagios team.
I found a forum topic that is describing the same problem as do have I. Traps from 2 NICs: https://support.nagios.com/forum/viewto ... 8&#p335481

I am also capturing all the udp packets comming to both our interfaces. But the traps routed to eth1 do not get processed.

root@lv-mon-p-master:~# tcpdump -vvvnn -i any src 10.232.235.11 or 10.12.22.35 and udp

This is the problem trap generated automatically and coming from the monitored vCenter to the interface eth1 (IP 10.230.5.240):
12:20:06.184129 IP (tos 0x0, ttl 62, id 3698, offset 0, flags [DF], proto UDP (17), length 240)
10.232.235.11.50550 > 10.230.5.240.162: [udp sum ok] { SNMPv1 { Trap(195) .1.3.6.1.4.1.6876.4.3 10.230.225.11 enterpriseSpecific s=203 1020073121 .1.3.6.1.4.1.6876.4.3.308.0=2 .1.3.6.1.4.1.6876.4.3.304.0="Yellow" .1.3.6.1.4.1.6876.4.3.305.0="Green" .1.3.6.1.4.1.6876.4.3.306.0="alarm.HostCPUUsageAlarm - Metric CPU Usage = 3%" .1.3.6.1.4.1.6876.4.3.307.0="10.230.220.44" } }

This one is my attempt to replicate the same trap from a linux command line and it comes from my testing CentOS vm which resides in the same network as interface eth0 (IP 10.12.22.11) on my Nagios server:
12:20:19.577380 IP (tos 0x0, ttl 64, id 1383, offset 0, flags [DF], proto UDP (17), length 239)
10.12.22.35.49628 > 10.12.22.11.162: [udp sum ok] { SNMPv1 { Trap(194) .1.3.6.1.4.1.6876.4.3 10.12.22.35 enterpriseSpecific s=203 284794559 .1.3.6.1.4.1.6876.4.3.308.0=2 .1.3.6.1.4.1.6876.4.3.304.0="Green" .1.3.6.1.4.1.6876.4.3.305.0="Yellow" .1.3.6.1.4.1.6876.4.3.306.0="alarm.HostCPUUsageAlarm - Metric CPU Usage = 10%" .1.3.6.1.4.1.6876.4.3.307.0="10.12.22.33" } }

But I can process only the latter, the one received by eth0 interface. It is processed by snmptrapd, logged into a defined file and passed to snmptt which logs it again and which passes it further to Nagios.


This is my actual /etc/sysconfig/snmptrapd setting:
OPTIONS="-On -tLf /var/log/snmptrapd.log"

I tried also to run it as /usr/sbin/snmptrapd -On -tLf /var/log/snmptrapd.log udp:127.0.0.1:162 udp:10.12.22.11:162 udp:10.230.5.240:162 -f

In debug mode /usr/sbin/snmptrapd -On -DALL -tLf /var/log/snmptrapd.log -f
- This is the only thing that looked like an error: "netsnmp_unix_transport: couldn't connect to "/var/agentx/master", errno 2 (No such file or directory)"

Neither this simple setting of /etc/snmp/snmptrapd.conf did help:
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler

Disabling firewall had no effect either.
Still no traps from eth1.


What else might interest you:
root@lv-mon-p-master:~# ss -ulpn | grep 162
UNCONN 0 0 *:162 *:* users:(("snmptrapd",pid=12048,fd=8))
root@lv-mon-p-master:~# ps -ef | grep trap
root 12048 1 0 13:54 ? 00:00:00 /usr/sbin/snmptrapd -On -tLf /var/log/snmptrapd.log -f
root@lv-mon-p-master:~# grep release /etc/*release
/etc/centos-release:CentOS Linux release 7.9.2009 (Core)
root@lv-mon-p-master:~# uname -a
Linux lv-mon-p-master 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux


Please, let me know how have you solved the ticket related to the "Traps from 2 NICs" forum thread.


Thanks
Jaroslav

[jsimek]

Update: There should be no problem in interface eth1, because today I managed to send a testing trap to it. This trap (my own, from a command line) was processed OK - by snmptrapd and snmptt.

Maybe there is a structural problem in the v1 traps from vCenter?

[ssax]

I'm going to lab this up and take a look, I'll send an update shortly.

[ssax]

I labbed this up and it's working on multiple NICs with EL7 with the same setup with v1 and v2c traps.

I used this in the /etc/sysconfig/snmptrapd:

Code: Select all

OPTIONS="-Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid udp:127.0.0.1:162 udp:192.168.X.X:162 udp:192.168.Y.Y:162"

What does your /var/log/snmptrapd.log show when the vcenter trap comes in?

Here is what mine shows:

Code: Select all

[root@xid ~]# ss -ulpn | grep 162
UNCONN     0      0      192.168.X.X:162                      *:*                   users:(("snmptrapd",pid=9126,fd=10))
UNCONN     0      0      192.168.Y.Y:162                      *:*                   users:(("snmptrapd",pid=9126,fd=9))
UNCONN     0      0      127.0.0.1:162                      *:*                   users:(("snmptrapd",pid=9126,fd=8))

[jsimek]

All my interfaces are listening on 162/udp:

Code: Select all

root@lv-mon-p-master:~# ss -ulpn | grep 162
UNCONN     0      0      192.168.1.1:162      *:*   users:(("snmptrapd",pid=6004,fd=11))
UNCONN     0      0      10.230.5.240:162     *:*   users:(("snmptrapd",pid=6004,fd=10))
UNCONN     0      0      10.12.22.11:162      *:*   users:(("snmptrapd",pid=6004,fd=9))
UNCONN     0      0      127.0.0.1:162        *:*   users:(("snmptrapd",pid=6004,fd=8))

Tcpdump in registering all of the incomming trap packets. But only the non-vCenter are recorded to /var/log/snmptrapd.log

[tgriep]

Try adding the following in the /etc/snmp/snmptrapd.conf file.

Code: Select all

snmpTrapdAddr udp:162,udp6:162

Save the change, restart snmptrapd and see if it works.

What is the IP address of the system that is sending the traps to the Nagios server?
Is it 10.232.235.11 or 10.12.22.35?

Run this on the Nagios server.

Code: Select all

tcpdump -i any -s 65535 -w 162.cap -n udp port 162

Force some traps from the remote vCenter server if possible, stop tcpdump and get the 162.cap file from the nagios server and upload it here.

Can you post the /var/log/snmptrapd.log file so we can view it?

Thank you.