snmp traps are being received, not processed.

[trel1234]

I have ensured that we have the proper dependencies, this was working previously but has stopped, I am seeing the traps are received at the interface, snmptrapd logs the traps, and by all accounts the traps are being received as they are showing up in the snmptt debug logs, however they do not progress past this point, they are not showing up in the snmptt logs in general, yet they do appear in the snmpd spool...

#
# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'

grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
grep -i -m 5 'exec' /etc/snmp/snmptt.conf
grep -i 'nag' /etc/group
grep -i 'snmp' /etc/group

ll /var/log/snmptt/
ll -d /var/log/snmptt/
ll /var/spool/snmptt
ll -d /var/spool/snmptt

yum list installed | grep -i snmp-rwxr-xr--. 1 root nagios 798 Aug 21 2019 addmib
-rwxr-xr--. 1 root root 2652 Aug 21 2019 snmptraphandling.py
-rwxr-xr--. 1 root root 30438 Aug 21 2019 snmpttconvertmib
# ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x 1 root root 31800 Feb 5 03:00 snmpd
-rwxr-xr-x 1 root root 31968 Feb 5 03:00 snmptrapd
-rwxr-xr-x. 1 root root 177466 Oct 23 2012 snmptt
-rwxr-xr-x. 1 root root 6493 Oct 23 2012 snmptthandler
#
# grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt
# grep -i -m 5 'exec' /etc/snmp/snmptt.conf
##EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
##EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
###EXEC qpage -f TRAP notifygroup1 "Link down on interface $1. Admin state: $2. Operational state: $3"
###EXEC qpage -f TRAP notifygroup1 "Link up on interface $1. Admin state: $2. Operational state: $3"
##EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"
# grep -i 'nag' /etc/group
nagios1000:nagios,apache,snmptt
nagcmd1001:nagios,apache,snmptt
# grep -i 'snmp' /etc/group
snmptt994:
nagios1000:nagios,apache,snmptt
nagcmd1001:nagios,apache,snmptt
#
# ll /var/log/snmptt/
total 2024064
-rwxrwxr-- 1 snmptt snmptt 21609187 Feb 21 10:35 snmptt.debug
-rwxrwxr-- 1 snmptt snmptt 430078501 Feb 21 09:28 snmptt.debug-20200221
-rwxrwxr-- 1 snmptt snmptt 11385111 Feb 21 10:35 snmptthandler.debug
-rwxrwxr-- 1 snmptt snmptt 25283161 Feb 21 03:12 snmptthandler.debug-20200221
-rwxrwxr-- 1 snmptt snmptt 0 Nov 23 03:10 snmptt.log
-rwxrwxr-- 1 snmptt snmptt 5900030 Nov 20 08:23 snmptt.log-20191123
-rwxrwxr-- 1 snmptt snmptt 0 Nov 23 03:10 snmpttsystem.log
-rwxrwxr--. 1 snmptt snmptt 8248 Nov 21 11:57 snmpttsystem.log-20191123
-rwxrwxr-- 1 snmptt snmptt 0 Nov 23 03:10 snmpttunknown.log
-rwxrwxr-- 1 snmptt snmptt 1577441650 Nov 21 11:57 snmpttunknown.log-20191123
# ll -d /var/log/snmptt/
drwxrwxr-x. 2 snmptt snmptt 4096 Feb 21 03:13 /var/log/snmptt/
# ll /var/spool/snmptt
total 0
-rw-r--r-- 1 root root 0 Feb 21 10:09 #snmptt-trap-1582241957462430
# ll -d /var/spool/snmptt
drwxrwxr-x. 2 snmptt snmptt 53248 Feb 21 10:35 /var/spool/snmptt
#
# yum list installed | grep -i snmp
net-snmp.x86_64 1:5.7.2-43.el7_7.3 @updates
net-snmp-agent-libs.x86_64 1:5.7.2-43.el7_7.3 @updates
net-snmp-devel.x86_64 1:5.7.2-43.el7_7.3 @updates
net-snmp-libs.x86_64 1:5.7.2-43.el7_7.3 @updates
net-snmp-perl.x86_64 1:5.7.2-43.el7_7.3 @updates
net-snmp-utils.x86_64 1:5.7.2-43.el7_7.3 @updates
perl-Net-SNMP.noarch 6.0.1-7.el7 @epel
perl-SNMP_Session.noarch 1.13-5.el7 @base
php-snmp.x86_64 5.4.16-46.el7 @base
snmptt.noarch 1.4-0.9.beta2.el7 @epel
#

[tgriep]

Can you run the following 2 commands as root.

Code: Select all

tar cvfz /tmp/snmp.tgz /etc/snmp/*
tar cvfz /tmp/sharesnmp.tgz /usr/share/snmp/mibs/* --dereference

Then post these 2 files so I can check the settings and the MIB files for any errors.

Code: Select all

/tmp/snmp.tgz
/tmp/sharesnmp.tgz

We would also need to know the OID or OID's that are getting receiveed so we can narrow down the configuration to a specific OID if needed.

Also, post an example entry from the snmptt.log, snmptt.debug of a received trap.

[tgriep]

Thanks for the files.
It looks like the config file that the SNMP Trap Interface component has need removed from the snmptt.ini file so none of the traps will be processed.
Edit the /etc/snmp/snmptt.ini file and change the bottom of the file from

Code: Select all

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
END

to

Code: Select all

snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
/etc/snmp/snmptt.conf.nxti
END

Save the change and restart snmptt

Code: Select all

service snmptt restart

Then send a trap to the XI server and see if it shows up on the Admin > Unconfigured Objects menu.
That is where to look for a new trap.

[trel1234]

I have updated the configuration file in order to pull in the nxti configuration file.

I began digging a little deeper into the logs and found that there initially was an issue with the perl module not being enabled, which prevented the OID's from resolving. I can also now see that all of the spool files for snmptt are blank and not updating.

Code: Select all

Trap not defined...

Sleeping for 5 seconds

Processing file: #snmptt-trap-1582241957462430
Reading trap.  Current time: Tue Feb 25 10:32:59 2020
  Invalid trap file.  Expected a serial time on the first line but got nothing
  Error processing trap file #snmptt-trap-1582241957462430.  Skipping...
Sleeping for 5 seconds

Processing file: #snmptt-trap-1582241957462430
Reading trap.  Current time: Tue Feb 25 10:33:04 2020
  Invalid trap file.  Expected a serial time on the first line but got nothing
  Error processing trap file #snmptt-trap-1582241957462430.  Skipping...

[tgriep]

Did you resolve the perl issue? If so, what was it as that may shed some light on the other issues?

Are all of the traps in the /var/spool/snmptt folder zero bytes?
Delete all of the traps from that folder and restart the following.

Code: Select all

service snmptrapd restart
service snmptt restart

Are the traps getting in to the /var/spool/snmptt folder valid now?
If so, are they getting processes by the snmptt daemon and removed from that folder?
Check the snmptt.log and the snmpttunknown.log file in the /var/log/snmptt folder.

If the traps are still zero bytes in the /var/spool/snmptt folder, enable logging for snmptrapd by following this article.
https://support.nagios.com/kb/article/s ... ce-88.html

[trel1234]

We are now receiving the traps.

So for clarification for anyone else who comes accross this thread with a similar issue:

After enabling the debugging for snmptt, the /var/log/snmptt/snmptt.debug log file began showing:

Code: Select all

Could not translate - Net-SNMP Perl module not enabled - will leave as-is

Reinstalling the Perl module for NET-SNMP solved that particular issue.

The logs indicating

Code: Select all

Invalid trap file.  Expected a serial time on the first line but got nothing

Required a complete clear of all of the spool in /var/spool/snmptt/

Then

Code: Select all

service snmptrapd restart
service snmptt restart

[scottwilkerson]

Great!

Thanks for sharing your info

Locking thread