Not receiving traps from Nagios 5.6.6

[dfmco]

I am receiving traps at the server:

Code: Select all

tcpdump -i ens192 -n -s0 -v port 162
tcpdump: listening on ens192, link-type EN10MB (Ethernet), capture size 262144 bytes
06:24:56.094311 IP (tos 0x0, ttl 252, id 4918, offset 0, flags [none], proto UDP (17), length 445)
    10.106.255.10.55084 > 10.106.156.102.snmptrap:  { SNMPv3 { F=ap } { USM B=3 T=164678676 U="nms" } { ScopedPDU [!scoped PDU]bd_62_47_fc_d6_2b_72_57_16_ea_4c_fc_a5_59_f9_41_fb_8f_dd_8b_84_c4_ca_ff_53_1e_d5_dc_f3_7c_e4_78_3b_39_1f_07_56_ce_3b_75_90_d2_2d_74_3a_82_ef_b3_b7_67_88_f3_f8_5c_26_5a_6d_65_c1_0f_57_58_b8_4f_81_16_08_58_de_70_a1_bb_ef_bf_06_e9_28_68_b4_4f_bb_c0_97_79_ea_2b_f0_8b_88_b6_39_63_ea_20_f6_c3_37_b6_86_04_4d_d4_71_cb_22_fc_68_cd_9b_10_e0_45_25_cd_44_07_fb_6b_a6_e6_bf_95_fb_fb_6f_ca_1c_3d_a3_f8_35_15_9e_72_4b_a0_22_3b_cb_39_98_1c_2f_ae_22_ba_60_de_5a_66_97_a6_4a_4e_f7_b4_3b_5b_a2_ba_9a_85_da_5d_2c_de_7c_48_77_c9_26_e9_12_de_4b_ee_09_2e_a6_6d_ae_34_a9_5a_6b_84_d4_9e_62_52_14_9a_c0_a5_20_90_a7_17_d8_10_69_dd_89_bc_1b_5a_54_41_e4_ca_8e_1a_f6_3e_b0_77_cb_12_f1_5b_97_8c_1e_de_6f_6d_dc_3b_2d_06_dd_0e_3c_7c_33_1f_a9_d7_24_11_e3_d6_e4_1c_d6_34_36_37_b7_2a_84_21_71_b0_3f_a3_a3_44_50_36_ee_6d_7b_8e_bd_cc_75_71_b4_fc_93_2f_24_3a_6a_4c_28_45_b5_cd_e4_f2_ec_af_b8_ee_fd_b4_e7_60_4b_5f_17_04_41_ea_45_96_67_44_93_29_3b_03_2a_75_84_59_23_7b_d7_0f_75_7a_a0_93_89_c8_d1_00_13_3e_d1_a1_a3_d3_8b_a4_fa_bf_96_ff_34_ed} } 
06:25:00.295004 IP (tos 0x0, ttl 255, id 13423, offset 0, flags [none], proto UDP (17), length 306)

snmptrapd and snmptt are running

Code: Select all

[root@camrn-harems-netmon-pri snmptt]# service snmptrapd status
Redirecting to /bin/systemctl status snmptrapd.service
● snmptrapd.service - Simple Network Management Protocol (SNMP) Trap Daemon.
   Loaded: loaded (/usr/lib/systemd/system/snmptrapd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-09-19 06:11:18 CDT; 24min ago
 Main PID: 19000 (snmptrapd)
   CGroup: /system.slice/snmptrapd.service
           └─19000 /usr/sbin/snmptrapd -Lsd -f

Sep 19 06:11:18 camrn-harems-netmon-pri systemd[1]: Starting Simple Network Management Protocol (SNMP) Trap Daemon....
Sep 19 06:11:18 camrn-harems-netmon-pri snmptrapd[19000]: NET-SNMP version 5.7.2
Sep 19 06:11:18 camrn-harems-netmon-pri systemd[1]: Started Simple Network Management Protocol (SNMP) Trap Daemon..
[root@camrn-harems-netmon-pri snmptt]# service snmptt status
Redirecting to /bin/systemctl status snmptt.service
● snmptt.service - SNMP Trap Translator (SNMPTT)
   Loaded: loaded (/usr/lib/systemd/system/snmptt.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-09-19 05:42:00 CDT; 53min ago
  Process: 21290 ExecStart=/usr/sbin/snmptt --daemon (code=exited, status=0/SUCCESS)
 Main PID: 21293 (snmptt)
   CGroup: /system.slice/snmptt.service
           ├─21292 /usr/bin/perl /usr/sbin/snmptt --daemon
           └─21293 /usr/bin/perl /usr/sbin/snmptt --daemon

Sep 19 05:42:00 camrn-harems-netmon-pri systemd[1]: Starting SNMP Trap Translator (SNMPTT)...
Sep 19 05:42:00 camrn-harems-netmon-pri systemd[1]: snmptt.service: Supervising process 21293 which is not our child. We'll most likely not notice when it exits.
Sep 19 05:42:00 camrn-harems-netmon-pri systemd[1]: Started SNMP Trap Translator (SNMPTT).

Firewall is open for UDP/162

Code: Select all

[root@camrn-harems-netmon-pri snmptt]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources: 
  services: dhcpv6-client http https ssh syslog
  ports: 80/tcp 443/tcp 22/tcp 7878/tcp 162/udp

snmptt.conf has directives for traps we should be recieving:

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Critical
FORMAT Link down on interface $1.  Admin state: $2.  Operational state: $3
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "LINK DOWN, $*"

SNMPv3 user and engine IDs are configured. Here is a sample (passwords redacted)

Code: Select all

traphandle default /usr/sbin/snmptthandler
#if using v3 (update user and password if required.  Must match router)
createUser uernameexample SHA authpassexample AES privpassexample
authUser log,execute,net uernameexample
createUser -e 0xEXAMPLEENGINEID uernameexample SHA authpassexample AES privpassexample

I followed this guide for further troubleshooting:
https://support.nagios.com/kb/article.p ... ategory=55

I enabled logging but the file was not created so I touched it but nothing is showing up. Why is that happening?

I ran debug output and you can see a trap from one of the hosts (10.106.255.1) in the output (file attached). It appears not to be matching the engineID but I have confirmed that it is correct. I am also concerned about the authpriv error:
snmp_parse: Parsed SNMPv3 message (secName:nms, secLevel:authPriv): ASN.1 parse error in message
Maybe that is only showing because the engine ID is not matching.

I have had a very similar issue before:
Not receiving SNMP Traps from Nagios
Postby dfmco » Fri May 20, 2016 1:32 pm

I wound up rebuilding from scratch to correct the problem back in 2016 but since this is happening again, I don't think it is a fluke and would like to figure out why this is failing.

[tgriep]

The log file may not of been creates or added to because of a permission issue with the file of folder where the log file is stored so check that.
The snmptt user and group needs to be able to write to the file.

Try setting the permissions for the file.

Your example snmptrapd.conf file, you show 2 create user lines but one user in the authuser line, is that how it is setup?

I would remove the redundant create user line to see if that gets things working.

When the snmptrapd daemon receives a trap, it stores it in this folder

Code: Select all

/var/spool/snmptt/

Do you see them there?

One thing I found working on some systems using SNMPv3, sometimes the snmptrapd daemon will not log anything in authentication errors.
It just silently drops the received data.

[dfmco]

Sorry if I was not clear but I ran through the troubleshooting document and got stuck on the debug.

No, there are no files created in /var/spool/snmptt/ so I continued on with the troubleshooting until the debug section.

There may be a misunderstanding on how this is intended to work and what I have set up.

Here is my snmptrapd.conf:
traphandle default /usr/sbin/snmptthandler
#if using v3 (update user and password if required. Must match router)
createUser exampleuser SHA exampleauthpass AES exampleprivpass
authUser log,execute,net exampleuser
createUser -e 0x8123456789012345678 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345679 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345670 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345671 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345672 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345673 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345674 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345675 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345676 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345677 exampleuser SHA exampleauthpass AES exampleprivpass

I thought this line was required but per your post, I am wondering if it should be there:
createUser exampleuser SHA exampleauthpass AES exampleprivpass

Because our environment is fairly large with multiple servers covering multiple locations, it would be a logistical nightmare to use unique users and passwords for the several hundred devices we manage. This would also make all of our automation more difficult. Because of this, we use the same user/pass per environment for multiple devices. I have never had an issue with any monitoring system doing this so I am a bit confused on the redundant user line that you pointed out but maybe I am just missing something. If I have 10 network devices I should be able to configure the same user/pass on all 10 using the unique engine id for each and snmptrapd should parse through the list until there is a match, correct? In other words, it receives the traps and goes through the list below from #1 to #10 until the engineid/user/auth/priv matches. This is how all of our other servers are set up and working.

createUser -e 0x8123456789012345678 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345679 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345670 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345671 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345672 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345673 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345674 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345675 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345676 exampleuser SHA exampleauthpass AES exampleprivpass
createUser -e 0x8123456789012345677 exampleuser SHA exampleauthpass AES exampleprivpass

I was a bit concerned that since the file was created in Windows notepad++ that maybe it was a line ending problem but I can't figure out how to check in VIM.

I tried removing all createUser lines EXCEPT line 1 and I sent a trap from there and /var/spool/snmptt does not get any files (snmptt is stopped).
traphandle default /usr/sbin/snmptthandler
#if using v3 (update user and password if required. Must match router)
#createUser exampleuser SHA exampleauthpass AES exampleprivpass
authUser log,execute,net exampleuser
createUser -e 0x8123456789012345678 exampleuser SHA exampleauthpass AES exampleprivpass

I also tried adding the createUser line without the engine ID like this:
traphandle default /usr/sbin/snmptthandler
#if using v3 (update user and password if required. Must match router)
createUser exampleuser SHA exampleauthpass AES exampleprivpass
authUser log,execute,net exampleuser
createUser -e 0x8123456789012345678 exampleuser SHA exampleauthpass AES exampleprivpass

That failed to generate files as well.

[tgriep]

I was just trying to see if the differing entries are causing conflicts so that is why I mentioned it.

I am thinking that the issue is that the devices that are sending the Traps are not using the correct Engine ID as I found this in your log file.

usm: no match on engineID (80 00 00 09 03 00 2C 54 2D 31 E9 00 )

Verify that the device is sending the correct Engine ID or, to troubleshoot this, simplify the configs by removing all of the createuser lines and remove the engine ID to see if that works.

Also, make sure the net-snmp package and it's dependencies are up to date on the server.

The link below is Net-SNMP guide for setting up snmptrapd for version 3 and it has some tests that you can run to see if you can get the traps received.
http://net-snmp.sourceforge.net/wiki/in ... _TRAP_User

I followed the examples and the snmptrapd daemon did receive the traps.

[dfmco]

That is the correct entry. Can I PM you the un-redacted file? That may help but I can not post it in a public forum due to security concerns.

To be clear, we have several other systems set up in the same way that have no issues with SNMPv3 traps.

This is an RPM install so we should be good on net-snmp, right?

Code: Select all

Package 1:net-snmp-5.7.2-43.el7.x86_64 already installed and latest version

Are you sure this is the correct file and path for the net-snmp user? I have never done this before in any of our currently working systems.

Code: Select all

"/var/net-snmp/snmptrapd.conf" [New DIRECTORY]

I want to make sure this does not cause an issue with updates. Can you validate that part of the instructions for me?
I checked on another box and it looks like we SHOULD NOT be editing this file which I found in a different location:

Code: Select all

cat /var/lib/net-snmp/snmptrapd.conf
#
# net-snmp (or ucd-snmp) persistent data file.
#
############################################################################
# STOP STOP STOP STOP STOP STOP STOP STOP STOP 
#
#          **** DO NOT EDIT THIS FILE ****
#
# STOP STOP STOP STOP STOP STOP STOP STOP STOP 
############################################################################
#
# DO NOT STORE CONFIGURATION ENTRIES HERE.
# Please save normal configuration tokens for snmptrapd in SNMPCONFPATH/snmptrapd.conf.
# Only "createUser" tokens should be placed here by snmptrapd administrators.
# (Did I mention: do not edit this file?)
<SNIP>

I also wanted to confirm if this line needs to be deleted or not:

Code: Select all

disableAuthorization yes

I did a stare and compare and my other files do not have spaces on this line. I corrected that.

Code: Select all

authUser log, execute, net nms

changed to:

Code: Select all

authUser log,execute,net nms

[tgriep]

Following the example from the net-snmp link, I created an /etc/snmp/snmptrapd.conf file like the following and the test traps I sent worked.

Code: Select all

traphandle default /usr/sbin/snmptthandler
createUser -e 0x8000000001020304 traptest SHA mypassword AES mypassword
authuser log traptest

And the correct file and path for the above is the /etc/snmp/snmptrapd.conf file.

This line does need to be removed from the snmptrapd.conf file.

Code: Select all

disableAuthorization yes

I don't think the spaces in the authUser option matter.
One thing to try, I know there is a limit that the passwords have to be 8 characters or greater, maybe the username has the same restriction. Make it longer and see if it works.

The following list is the net-snmp packages that is installed on my test system that worked running the test snmpv3 tests.

Code: Select all

net-snmp.x86_64                         1:5.7.2-43.el7                 @base
net-snmp-agent-libs.x86_64              1:5.7.2-43.el7                 @base
net-snmp-devel.x86_64                   1:5.7.2-43.el7                 @base
net-snmp-libs.x86_64                    1:5.7.2-43.el7                 @base
net-snmp-perl.x86_64                    1:5.7.2-43.el7                 @base
net-snmp-utils.x86_64                   1:5.7.2-43.el7                 @base

I could not find very many details on the /var/lib/net-snmp/snmptrapd.conf file.
I think it is used by the daemon to store the username, encrypted password, etc that is associated to the entry in the /etc/snmp/snmptrapd.conf as I found this in it.

Code: Select all

usmUser 1 3 0x8000000001020304 "traptest" "traptest" NULL .1.3.6.1.6.3.10.1.1.3 0xd845017af8ec5da84d935b40fa124f20c256826c .1.3.6.1.6.3.10.1.2.4 0xd845017af8ec5da84d935b40fa124f20 ""
engineBoots 1
oldEngineID 0x80001f888088a1b95e8cd9845d00000000

[dfmco]

We use the nms username on all managed systems so that one is OK.

Can we get this to development to check for a bug? It seems that we have run out of road on the troubleshooting.

I removed the disableauthorization line but still no joy (it exists on all other installs that work). Logs show it not finding the engine id even though I can clearly see it in the conf file.

Please let me know next steps. We are in production now so I need to get this fixed ASAP.

Thanks!

[tgriep]

If you want to file a bug report, you can use the following link to create one.
http://www.net-snmp.org/support/bugreports.html

Did you try the troubleshooting steps from this link by sending the test traps using the snmptrap command using the EngineID, username and password from the snmptrapd.conf file?
http://net-snmp.sourceforge.net/wiki/in ... _TRAP_User

[dfmco]

Yes I did when you asked. No joy. Something is not right.

[ssax]

Do v2 traps even show up now in /var/spool/snmptt?

Please send the output of this command:

Code: Select all

ls -l /usr/sbin/snmptthandler

Please run these commands (as root/sudo) and PM me the resulting /tmp/SNMPFILES.zip file:

Code: Select all

zip -r /tmp/SNMPFILES.zip /etc/snmp /usr/lib/systemd/system/snmptrapd.service

Please include the file you referenced before, you can encrypt and PM all the info then send another PM with the pass if you'd like.

Thank you!