snmp trap not being processed

[dslaughter]

nagiosxi 5.6.8
centos 7.6.1810

I've successfully setup and been using snmp traps with nagiosxi. I have an application that sends traps. These traps are not making it to nagiosxi.

The application server sending the traps is on a different server than the nagiosxi server. I can send a netSnmpExampleHeartbeatNotification test trap from the application server to nagiosxi and it is processed by nagiosxi, shows in logs and received traps list so the trap configuration seems correct. Using tcpdump on the application server I see a trap is sent. Using tcpdump on nagiosxi server I see the trap come in. It does not make it to received trap list, snmpttunknown.log or snmptt.log. I also checked corelog.newobjects and I didn't see anything from the application server.

The trap was defined by importing the mib.

lostTarget .1.3.6.1.4.1.30359.1.1.4.2.2.2002 Status Events Normal Communication to the target has been lost Variables: 1: majorId 2: majorDesc 3: majorAction 4: majorSeverity 5: majorNode 6: majorComponent 7: majorContext 8: majorTimestamp 9: majorDomain

Here is tcpdump output on nagiosxi server

21:17:14.798091 IP acc-1-1.health-polling > nms-1.snmptrap: C="ahjtv4$" V2Trap(425) system.sysUpTime.0=200835436 S:1.1.4.1.0=E:30359.1.1.4.2.2.2002 76="3.8.12.70" E:30359.1.1.4.2.21="2002" E:30359.1.1.4.2.22="Communication to the target has been lost" E:30359.1.1.4.2.23="Please verify target status or network connectivity. " E:30359.1.1.4.2.24="MAJOR" E:30359.1.1.4.2.25="acc-1-2" E:30359.1.1.4.2.26="mgmt1" E:30359.1.1.4.2.27="prov2" E:30359.1.1.4.2.28="19-12-05 21:17:12.689" E:30359.1.1.4.2.29="acc-1"

I'm not sure what to do next to troubleshoot. Can you provide some direction?

[tgriep]

We would need to see the SNMP configuration files and the MIB files from the server so can you run the following 3 commands as root.

Code: Select all

tar cvfz /tmp/snmp.tgz /etc/snmp/*
tar cvfz /tmp/sharesnmp.tgz /usr/share/snmp/mibs/* --dereference
tar cvfz /tmp/snmplog.tgz /var/log/snmptt/*

Then post these 3 files so I can check the settings and the MIB files for any errors.

Code: Select all

/tmp/snmp.tgz
/tmp/sharesnmp.tgz
/tmp/snmplog.tgz

[dslaughter]

Here ya go...

[tgriep]

The configs look OK for that OID so I will need you to run the following as root and post the /tmp/info.txt file to continue on.

Code: Select all

ls -l /var/spool/snmptt/ >/tmp/info.txt
ps -ef >>/tmp/info.txt
iptables -L >>/tmp/info.txt

Make sure the firewall is enabled to allow inbound traffic on UDP port 162.
Make sure the snmptrapd daemon is running as well.

Thanks

[dslaughter]

Here is the info requested.

I'm receiving traps from other servers successfully. They show in received traps and are sent to the correct service for the correct server. So the firewall config seems ok. Don't you mean 162 udp in bound? Remember the trap is received on the nagiosxi server it just doesn't get processed by nagiosxi.

public (active)
target: default
icmp-block-inversion: no
interfaces: ens192 ens224
sources:
services: ssh dhcpv6-client mountd rpc-bind nfs snmp snmptrap
ports: 80/tcp 443/tcp 22/tcp 7878/tcp 162/udp 5667/tcp
protocols:
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:


root 6345 1 0 Aug27 ? 00:01:40 /usr/sbin/snmptrapd -Ln -f
root 24639 32211 0 11:46 pts/0 00:00:00 grep --color=auto snmptrap

[dslaughter]

I was able to send a test trap using snmptrap command on command line. It looks like all the trap vars are not coming in from the application. I included all vars from the mib and It worked. That's good but shouldn't nagios throw a trap with a bad format into one of the logs?

I need to work with the application team to get this completely sorted. I would like to keep this issue open for at least another day while I collect more information.

[tgriep]

Typo on my part, it is UDP port 162, I updated the post with the correct value.

The snmptrapd daemon first receives the trap and puts it in the spool folder for the snmptt daemon.
If the trap is not formatted correctly, the snmptrapd daemon may just delete it so the snmptt daemon may never see it.

In this KB article, it had documentation in enabling logging dor the snmptrapd daemon. Enable it and see if the traps are getting dropped.
https://support.nagios.com/kb/article/s ... ce-88.html

[dslaughter]

Go ahead and close this thread. I'm going to start the logging of snmptrapd. If I have any issues I'll start a new thread.

[tgriep]

OK, I'll close it for you.