Nagios Support Forum

Hi, I am setting up Nagios XI for receiving snmp traps for our server.
I can see the trap received from the service Status :

SNMP Traps
Passive Only Check
Ok 14m 58s 1/1 2023-10-03 17:19:48 The LMR-IWF Fault Manager (FM) Server will clear an Alarm 07 E7 0A 03 16 13 30 00 communicationsAlarm E_HC_CDR lmr-iwf-3-gp1 Monitor 1 3.1.0 CdrMonitor major The CDR Server health check has failed. {"error": "HTTPConnectionPool(host='127.0.0.1', port=9082): Max retries exceeded with url: /metrics (Caused by NewConnectionError('urllib3.connection.HTTPConnection object at 0x7fc1e8751df0: Failed to establish a new connection: Errno 111 Connection refused'))", "api": "CDR Server"} applicationSubsystemFailure The IWF Monitor application is sending health checks to the CDR server. The IWF Monitor considers the CDR server to be unreachable. e155365b-f8f8-4996-a6e1-cb8bc30800c6

But from the tab of Received Traps under SNMP Trap Interface, I cannot find this trap. I can see the trap received from /var/spool/snmptt like below.
mcptt@iwf-ancn13-ubuntu:/var/spool/snmptt$ cat '#snmptt-trap-1696370246972538'
1696370246
lmr-iwf-3-gp1
UDP: [10.20.44.65]:44799->[172.22.106.25]:162
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:00:08.63
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-SMI::enterprises.17869.3.1.2.1.3
SNMPv2-SMI::enterprises.17869.3.1.2.2.1 "07 E7 0A 03 15 39 1B 00 "
SNMPv2-SMI::enterprises.17869.3.1.2.2.2 2
SNMPv2-SMI::enterprises.17869.3.1.2.2.3 "E_HC_CDR"
SNMPv2-SMI::enterprises.17869.3.1.2.2.4 "lmr-iwf-3-gp1"
SNMPv2-SMI::enterprises.17869.3.1.2.2.5 "Monitor"
SNMPv2-SMI::enterprises.17869.3.1.2.2.6 "1"
SNMPv2-SMI::enterprises.17869.3.1.2.2.7 "3.1.0"
SNMPv2-SMI::enterprises.17869.3.1.2.2.8 "CdrMonitor"
SNMPv2-SMI::enterprises.17869.3.1.2.2.9 1
SNMPv2-SMI::enterprises.17869.3.1.2.2.10 "The CDR Server health check has failed."
SNMPv2-SMI::enterprises.17869.3.1.2.2.11 "{\"api\": \"CDR Server\", \"error\": \"HTTPConnectionPool(host='127.0.0.1', port=9082): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc1e97d4e80>: Failed to establish a new connection: [Errno 111] Connection refused'))\"}"
SNMPv2-SMI::enterprises.17869.3.1.2.2.12 3
SNMPv2-SMI::enterprises.17869.3.1.2.2.13 "The IWF Monitor application is sending health checks to the CDR server. The IWF Monitor considers the CDR server to be unreachable."
SNMPv2-SMI::enterprises.17869.3.1.2.2.14 "17cd7204-a66e-42bd-87db-86b50a4037d8"

Read the snmp log files.
Check unconfigured objects on the admin tab.

There is no log within the time of the trap.

mcptt@iwf-ancn13-ubuntu:/var/log$ cd snmptt
mcptt@iwf-ancn13-ubuntu:/var/log/snmptt$ ll
total 16
drwxr-xr-x 2 Debian-snmp snmptt 59 Oct 1 00:00 ./
drwxr-xr-x 21 root syslog 4096 Oct 1 00:00 ../
-rw-r--r-- 1 root root 6784 Oct 3 14:52 snmpttsystem.log
-rw-r--r-- 1 root root 716 Sep 29 14:10 snmpttsystem.log.1.gz
mcptt@iwf-ancn13-ubuntu:/var/log/snmptt$ cat snmpttsystem.log

Also the Unconfigured Objects does not show the host that receiving the trap. If I sent the test snmp trap, it will show in the Received Traps tab.

Just found out that there were over 10K entries in the Received Traps tab. Those are the test traps we tested earlier. We just sent like 10 of them but it repeats endlessly until the buffer full so new traps are not shown there.

I deleted them all manually. Then I can see new traps on the Received Traps tab. But one single trap will repeat multiple times.
Also is it normal that the trap is kept in the /var/spool/snmptt folder?

Hi @davidvu, thanks for reaching out.

Are your received traps showing under the same event name? SNMPTT allows multiple events to use the same OID, in which case you'll see multiple entries show up for the same incoming event.

It doesn't sound normal to be to have traps stay in the `/var/spool/snmptt` folder. What are the permissions/ownership on the folder? Does `snmptt:snmptt` have access?

Thanks for replying:

Below the permission for /var/spool/snmptt

root@iwf-ancn13-ubuntu:/var/spool# ls -lrt
total 12
drwx------ 2 syslog adm 6 Dec 30 2021 rsyslog
drwxr-xr-x 3 root root 22 Aug 7 17:53 cron
drwx--x--- 3 root lp 17 Aug 7 17:53 cups
drwxr-xr-x 3 root root 26 Aug 7 17:54 libreoffice
lrwxrwxrwx 1 root root 7 Sep 13 23:27 mail -> ../mail
drwxr-xr-x 2 root root 63 Sep 13 23:31 anacron
drwxr-xr-x 18 root root 4096 Oct 4 11:45 postfix
drwxrws--- 2 smmsp smmsp 6 Oct 4 16:30 mqueue-client
drwxr-s--- 2 smmta smmsp 6 Oct 4 16:30 mqueue
drwxrwxr-x 3 Debian-snmp snmptt 4096 Oct 4 16:31 snmptt

Should it set as snmptt:snmptt?

Also I cannot see any log from /var/log/snmp after the trap received.

root@iwf-ancn13-ubuntu:/var/spool# ls -lrt snmptt
snmptt:
total 20
drwxr-xr-x 2 root root 8192 Oct 4 16:31 tmp
-rw-r--r-- 1 Debian-snmp Debian-snmp 1320 Oct 4 16:31 '#snmptt-trap-1696455075469542'
-rw-r--r-- 1 Debian-snmp Debian-snmp 1322 Oct 4 16:31 '#snmptt-trap-1696455114028227'
root@iwf-ancn13-ubuntu:/var/spool# ls -lrt /var/log/snmptt/
total 16
-rw-r--r-- 1 root root 716 Sep 29 14:10 snmpttsystem.log.1.gz
-rw-r--r-- 1 root root 8904 Oct 4 16:30 snmpttsystem.log

Regarding the missing logs in /var/log/snmptt/, it could be due to the permissions issue. After correcting the ownership, you should monitor the logs for any additional drift hunters information. If there are still issues, you might want to check the Snmptt configuration (snmptt.ini) to ensure that logging is properly configured.