SNMP Trap Not Working

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Locked
loopie
Posts: 5
Joined: Wed Mar 26, 2014 5:26 pm

SNMP Trap Not Working

Post by loopie »

Pretty new to the SNMP Trap thing and doing a bunch of forum and google searching.
Still having problems getting traps to show up in the XI webpages.
Here is a review of what I have so far..

Versions installed

Code: Select all

$  rpm -qa | grep snmp
php-snmp-5.3.3-27.el6_5.x86_64
net-snmp-utils-5.5-49.el6_5.1.x86_64
net-snmp-libs-5.5-49.el6_5.1.x86_64
net-snmp-perl-5.5-49.el6_5.1.x86_64
snmptt-1.4-0.9.beta2.el6.noarch
net-snmp-5.5-49.el6_5.1.x86_64

Code: Select all

$  ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root   nagios      804 Feb 18 14:41 addmib
-rwxr-xr-x   1 root   root       2078 Feb 18 14:41 snmptraphandling.py
-rwxr-xr-x   1 root   root      30438 Feb 18 14:41 snmpttconvertmib

Code: Select all

$ sudo cat /etc/snmp/snmptrapd.conf
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler

Code: Select all

$ grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt

Code: Select all

$ grep -i -m 5 'exec' /etc/snmp/snmptt.conf
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"

Code: Select all

$ grep -i 'nag' /etc/group
nagios:x:500:nagios,apache,snmptt
nagcmd:x:501:nagios,apache,snmptt

Code: Select all

$ grep -i 'snmp' /etc/group
nagios:x:500:nagios,apache,snmptt
nagcmd:x:501:nagios,apache,snmptt
snmptt:x:496:

Code: Select all

$ ll /var/log/snmptt/
total 31212
-rw-r--r-- 1 snmptt root 8699898 Mar 26 15:38 snmptt.debug
-rw-r--r-- 1 snmptt root 8708115 Mar 26 03:15 snmptt.debug-20140326
-rw-r--r-- 1 root   root 1067351 Mar 26 15:35 snmptthandler.debug
-rw-r--r-- 1 root   root 1065975 Mar 26 03:10 snmptthandler.debug-20140326
-rw-rw-r-- 1 snmptt root 1689095 Mar 26 15:35 snmptt.log
-rw-rw-r-- 1 snmptt root 3838760 Mar  9 03:15 snmptt.log-20140309
-rw-rw-r-- 1 snmptt root 3381852 Mar 16 04:45 snmptt.log-20140316
-rw-rw-r-- 1 snmptt root 2576782 Mar 21 14:20 snmptt.log-20140321
-rw-rw-r-- 1 snmptt root  741794 Mar 23 03:30 snmptt.log-20140323
-rw-r--r-- 1 root   root       0 Mar 26 03:15 snmpttsystem.log
-rw-r--r-- 1 root   root     238 Feb 18 14:41 snmpttsystem.log-20140223
-rw-r--r-- 1 root   root    1075 Feb 25 08:38 snmpttsystem.log-20140302
-rw-r--r-- 1 root   root    1890 Mar 20 16:40 snmpttsystem.log-20140321
-rw-r--r-- 1 root   root    1260 Mar 25 14:44 snmpttsystem.log-20140326
-rw-rw-r-- 1 snmptt root   14073 Mar 25 15:03 snmpttunknown.log
-rw-rw-r-- 1 snmptt root   16675 Mar  9 01:59 snmpttunknown.log-20140309
-rw-rw-r-- 1 snmptt root   41326 Mar 16 03:32 snmpttunknown.log-20140316
-rw-rw-r-- 1 snmptt root    8498 Mar 20 10:30 snmpttunknown.log-20140321
-rw-rw-r-- 1 snmptt root    3826 Mar 23 03:32 snmpttunknown.log-20140323

Code: Select all

$ ll -d /var/log/snmptt/
drwxrwxr-x 2 snmptt snmptt 4096 Mar 26 03:15 /var/log/snmptt/

Code: Select all

$ ll /var/spool/snmptt/
total 0

Code: Select all

$ ll -d /var/spool/snmptt/
drwxrwxr-x 2 snmptt snmptt 4096 Mar 26 15:35 /var/spool/snmptt/
Client Server

Code: Select all

$ sudo snmptrap -v 1 -c public nagios .1.3.6.1.6.3 "" 0 0 coldStart.0
$
Back to nagios server:

Code: Select all

$ tail -30 snmptthandler.debug

SNMPTTHANDLER started: Wed Mar 26 15:41:08 2014
s = 1395873668, usec = 127704
s_pad = 1395873668, usec_pad = 127704

Data received:
syslog.ourcompany.com
UDP: [10.14.134.21]:51522->[10.14.134.14]
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:00:00.00
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-MIB::coldStart
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 10.14.134.21
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 SNMPv2-SMI::snmpModules

$ tail snmptt.log
Wed Mar 26 15:41:05 2014 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" syslog - Device reinitialized (coldStart)
Wed Mar 26 15:41:08 2014 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" syslog - Device reinitialized (coldStart)
# Here where the fun starts and I get confused. How does the translation happen?

Code: Select all

$ tail -f  snmptt.debug

Sleeping for 5 seconds

Sleeping for 5 seconds


Processing file: #snmptt-trap-1395873966150083
Reading trap.  Current time: Wed Mar 26 15:46:07 2014
Symbolic trap variable name detected (DISMAN-EVENT-MIB::sysUpTimeInstance).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.2.1.1.3.0
Symbolic trap variable name detected (SNMPv2-MIB::snmpTrapOID.0).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3.1.1.4.1.0
Symbolic trap variable name detected (SNMP-COMMUNITY-MIB::snmpTrapAddress.0).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3.18.1.3.0
Symbolic trap variable name detected (SNMP-COMMUNITY-MIB::snmpTrapCommunity.0).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3.18.1.4.0
Symbolic trap variable name detected (SNMPv2-MIB::snmpTrapEnterprise.0).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3.1.1.4.3.0
Raw trap passed from snmptrapd:
1395873966
syslog.ourcompany.com
UDP: [10.14.134.21]:54350->[10.14.134.14]
DISMAN-EVENT-MIB::sysUpTimeInstance 0:0:00:00.00
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-MIB::coldStart
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 10.14.134.21
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 SNMPv2-SMI::snmpModules
Items passed from snmptrapd:
value 0: syslog.ourcompany.com
value 1: 10.14.134.21
value 2: .1.3.6.1.2.1.1.3.0
value 3: 0:0:00:00.00
value 4: .1.3.6.1.6.3.1.1.4.1.0
value 5: SNMPv2-MIB::coldStart
value 6: .1.3.6.1.6.3.18.1.3.0
value 7: 10.14.134.21
value 8: .1.3.6.1.6.3.18.1.4.0
value 9: public
value 10: .1.3.6.1.6.3.1.1.4.3.0
value 11: SNMPv2-SMI::snmpModules
Symbolic trap variable name detected (SNMPv2-MIB::coldStart).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3.1.1.5.1
Symbolic trap variable name detected (SNMPv2-SMI::snmpModules).  Will attempt to translate to a numerical OID
  Translated to .1.3.6.1.6.3
Agent IP address (10.14.134.21) is the same as the host IP, so copying the host name: syslog.ourcompany.com
Trap received from syslog.ourcompany.com: SNMPv2-MIB::coldStart
0:              hostname
1:              ip address
2:              uptime
3:              trapname / OID
4:              ip address from trap agent
5:              trap community string
6:              enterprise
7:              securityEngineID        (snmptthandler-embedded required)
8:              securityName            (snmptthandler-embedded required)
9:              contextEngineID         (snmptthandler-embedded required)
10:             contextName             (snmptthandler-embedded required)
0+:             passed variables
Value 0: syslog
Value 1: 10.14.134.21
Value 2: 0:0:00:00.00
Value 3: .1.3.6.1.6.3.1.1.5.1
Value 4: 10.14.134.21
Value 5: public
Value 6: .1.3.6.1.6.3
Value 7:
Value 8:
Value 9:
Value 10:
Agent dns name: syslog
Exact match of trap found in EVENT hash table
Working with EVENT entry: .1.3.6.1.6.3.1.1.5.1 => coldStart,Status Events,Normal,
  No nodes defined for this entry so all nodes will match
  No MATCH entries defined for this entry
Trap defined, processing...
PREEXEC line(s):
FORMAT line:
OID of enterprise: .1.3.6.1.6.3.  Will attempt to translate to text
    OID found in cache:  '.1.3.6.1.6.3' -> 'snmpModules'
  Translated to snmpModules
OID of received trap: .1.3.6.1.6.3.1.1.5.1.  Will attempt to translate to text
    OID found in cache:  '.1.3.6.1.6.3.1.1.5.1' -> 'coldStart'
  Translated to coldStart
Device reinitialized (coldStart)
.1.3.6.1.6.3.1.1.5.1 Normal "Status Events" syslog - Device reinitialized (coldStart)
EXEC line(s):
  EXEC line not defined
Sleeping for 5 seconds
It looks like the system accepts the trap, then the translator cannot process it.
The client machine is already running nrpe client and is registered and showing up in the Nagios XI interface properly.
I have setup the SNMP Traps config for the client, but Nagios show "waiting for traps".

Code: Select all

Host   	Service	Status	Duration	Attempt          Last Check	Status Information
 	
syslog       SNMP Traps   Passive Only Check	Ok 	1d 3h 30m 38s 	1/1 	2014-03-25 12:23:56 	Waiting for trap...
What step did I forget?

Thanks,
Loopie
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: SNMP Trap Not Working

Post by lmiltchev »

Run the following command and show us the output:

Code: Select all

grep "net_snmp_perl_enable = " /etc/snmp/snmptt.ini
Be sure to check out our Knowledgebase for helpful articles and solutions!
loopie
Posts: 5
Joined: Wed Mar 26, 2014 5:26 pm

Re: SNMP Trap Not Working

Post by loopie »

$ grep "net_snmp_perl_enable = " /etc/snmp/snmptt.ini
net_snmp_perl_enable = 1
loopie
Posts: 5
Joined: Wed Mar 26, 2014 5:26 pm

Re: SNMP Trap Not Working

Post by loopie »

More info.. I am keeping at this!!
Created TRAP-TEST-MIB.txt and put it in the /usr/share/snmp/mibs/ dir.
_________________________________________
TRAP-TEST-MIB DEFINITIONS ::= BEGIN
IMPORTS ucdExperimental FROM UCD-SNMP-MIB;

demotraps OBJECT IDENTIFIER ::= { ucdExperimental 990 }

demo-trap TRAP-TYPE
STATUS current
ENTERPRISE demotraps
VARIABLES { sysLocation }
DESCRIPTION "This is just a demo"
::= 17

END
________________________________________
Converted it to snmptt format..
sudo snmpttconvertmib --in=/usr/share/snmp/mibs/TRAP-TEST-MIB.txt --out=/etc/snmp/snmptt.conf --debug --exec='/usr/share/nagios3/plugins/eventhandlers/submit_check_result $r TRAP 1'

Total translations: 1
Successful translations: 1
Failed translations: 0

Checked /etc/snmp/snmptt.conf for output and found...
______________
.
.
.
MIB: TRAP-TEST-MIB (file:/usr/share/snmp/mibs/TRAP-TEST-MIB.txt) converted on Thu Mar 27 14:19:25 2014 using snmpttconvertmib v1.4beta2
#
# Lots of stuff here from other converted mibs.. :-)
#
EVENT demo-trap .1.3.6.1.4.1.2021.13.990.0.17 "Status Events" Normal
FORMAT This is just a demo $*
EXEC /usr/share/nagios3/plugins/eventhandlers/submit_check_result $r TRAP 1 "This is just a demo $*"
SDESC
This is just a demo
Variables:
1: sysLocation
EDESC
_________________________
Looks like a single OID has been defined. All looks good..

On remote host I run..
$ sudo snmptrap -v 1 -c public_password nagios_server TRAP-TEST-MIB::demotraps localhost 6 17 '' RFC1213-MIB::sysName.0 s "Dashang"
$
Runs OK.
Back on the Nagios server I watch the log files..
$ sudo tail -f /var/log/snmp/*.log
==> /var/log/snmptt/snmpttunknown.log <==
Thu Mar 27 14:26:24 2014: Unknown trap (.1.3.6.1.4.1.2021.13.990.0.17) received from client_server at:
Value 0: client_server
Value 1: 10.14.134.21
Value 2: 1:3:54:51.98
Value 3: .1.3.6.1.4.1.2021.13.990.0.17
Value 4: 127.0.0.1
Value 5: public
Value 6: .1.3.6.1.4.1.2021.13.990
Value 7:
Value 8:
Value 9:
Value 10:
Ent Value 0: .1.3.6.1.2.1.1.5.0=Dashang

==> /var/log/snmptt/snmpttsystem.log <==
--- No new output---

I do the same thing except for watching all the debug files.
$ sudo tail -f /var/log/snmp/*.debug
==> /var/log/snmptt/snmptthandler.debug <==
SNMPTTHANDLER started: Thu Mar 27 14:29:24 2014
s = 1395955764, usec = 945577
s_pad = 1395955764, usec_pad = 945577
Data received:
clientserver.mycompany.com
UDP: [10.14.134.21]:53670->[10.14.134.14]
DISMAN-EVENT-MIB::sysUpTimeInstance 1:3:57:51.70
SNMPv2-MIB::snmpTrapOID.0 UCD-SNMP-MIB::ucdExperimental.990.0.17
SNMPv2-MIB::sysName.0 Dashang
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 127.0.0.1
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 UCD-SNMP-MIB::ucdExperimental.990

==> /var/log/snmptt/snmptt.debug <==
Sleeping for 5 seconds

Processing file: #snmptt-trap-1395955764945577
Reading trap. Current time: Thu Mar 27 14:29:28 2014
Symbolic trap variable name detected (DISMAN-EVENT-MIB::sysUpTimeInstance). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.2.1.1.3.0
Symbolic trap variable name detected (SNMPv2-MIB::snmpTrapOID.0). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.6.3.1.1.4.1.0
Symbolic trap variable name detected (SNMPv2-MIB::sysName.0). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.2.1.1.5.0
Symbolic trap variable name detected (SNMP-COMMUNITY-MIB::snmpTrapAddress.0). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.6.3.18.1.3.0
Symbolic trap variable name detected (SNMP-COMMUNITY-MIB::snmpTrapCommunity.0). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.6.3.18.1.4.0
Symbolic trap variable name detected (SNMPv2-MIB::snmpTrapEnterprise.0). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.6.3.1.1.4.3.0
Raw trap passed from snmptrapd:
1395955764
client_server.mycompany.com
UDP: [10.14.134.21]:53670->[10.14.134.14]
DISMAN-EVENT-MIB::sysUpTimeInstance 1:3:57:51.70
SNMPv2-MIB::snmpTrapOID.0 UCD-SNMP-MIB::ucdExperimental.990.0.17
SNMPv2-MIB::sysName.0 Dashang
SNMP-COMMUNITY-MIB::snmpTrapAddress.0 127.0.0.1
SNMP-COMMUNITY-MIB::snmpTrapCommunity.0 "public"
SNMPv2-MIB::snmpTrapEnterprise.0 UCD-SNMP-MIB::ucdExperimental.990

Items passed from snmptrapd:
value 0: client_server.mycompany.com
value 1: 10.14.134.21
value 2: .1.3.6.1.2.1.1.3.0
value 3: 1:3:57:51.70
value 4: .1.3.6.1.6.3.1.1.4.1.0
value 5: UCD-SNMP-MIB::ucdExperimental.990.0.17
value 6: .1.3.6.1.2.1.1.5.0
value 7: Dashang
value 8: .1.3.6.1.6.3.18.1.3.0
value 9: 127.0.0.1
value 10: .1.3.6.1.6.3.18.1.4.0
value 11: public
value 12: .1.3.6.1.6.3.1.1.4.3.0
value 13: UCD-SNMP-MIB::ucdExperimental.990
Symbolic trap variable name detected (UCD-SNMP-MIB::ucdExperimental.990.0.17). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.4.1.2021.13.990.0.17
Symbolic trap variable name detected (UCD-SNMP-MIB::ucdExperimental.990). Will attempt to translate to a numerical OID
Translated to .1.3.6.1.4.1.2021.13.990
Agent IP address (127.0.0.1) resolved to: localhost.localdomain
Trap received from client_server.mycompany.com: UCD-SNMP-MIB::ucdExperimental.990.0.17
0: hostname
1: ip address
2: uptime
3: trapname / OID
4: ip address from trap agent
5: trap community string
6: enterprise
7: securityEngineID (snmptthandler-embedded required)
8: securityName (snmptthandler-embedded required)
9: contextEngineID (snmptthandler-embedded required)
10: contextName (snmptthandler-embedded required)
0+: passed variables

Value 0: client_server
Value 1: 10.14.134.21
Value 2: 1:3:57:51.70
Value 3: .1.3.6.1.4.1.2021.13.990.0.17
Value 4: 127.0.0.1
Value 5: public
Value 6: .1.3.6.1.4.1.2021.13.990
Value 7:
Value 8:
Value 9:
Value 10:
Agent dns name: localhost
Ent Value 0 ($1): .1.3.6.1.2.1.1.5.0=Dashang
Exact match of trap NOT found in EVENT hash table
Looking for wildcards in the EVENT hash table
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.1.2021.13.990.0.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.1.2021.13.990.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.1.2021.13.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.1.2021.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.1.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.4.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.1.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.6.*
Drilling down looking for wildcards in the EVENT hash table
.1.3.*
Drilling down looking for wildcards in the EVENT hash table
.1.*
Drilling down looking for wildcards in the EVENT hash table
.*
Trap not defined...
Sleeping for 5 seconds

------------------------------------------------------------
It seems I have the OID's defined and in the proper place, but they are not being captured/translated properly.
The debug file seems to show the proper OID's but the translation is not taking place.
Any ideas what I skipped/forgot?

thx,
Loopie..
loopie
Posts: 5
Joined: Wed Mar 26, 2014 5:26 pm

Re: SNMP Trap Not Working

Post by loopie »

Created a second mib.. NOTIFICATION-TEST-MIB.txt via google search..
Converted it OK, Sent a version 2 snmp trap to the Nagios server in an attempt to find out if Nagios only works with v2 type snmp traps.

From client_server:
$ sudo snmptrap -v 2c -c public nagios_server '' NOTIFICATION-TEST-MIB::demo-notif SNMPv2-MIB::sysLocation.0 s "just here"
$

On the nagios server I noted output of
$ sudo tail -f /var/log/snmptt/*.debug
__________________________________________________
.
.
.
OID of received trap: .1.3.6.1.4.1.2021.991.17. Will attempt to translate to text
OID found in cache: '.1.3.6.1.4.1.2021.991.17' -> 'ucdavis.991.17'
Translated to ucdavis.991.17
EXEC command:/usr/share/nagios3/plugins/eventhandlers/submit_check_result client_server TRAP 1 "Just a test notification just here"
Sleeping for 5 seconds

_____________________________________
Checked the /usr/share/nagios3/plugins directory and there is no eventhandlers dir and in searching the system I don't have a "submit_check_result" script anywhere on the system. It maybe the previous tech didn't get the complete Nagios XI install done or is missing one or more modules.
I don't see where the eventhandlers directory is specified, but the config shows the eventhandlers are enabled. Hmm..
Help!
Loopie..
slansing
Posts: 7698
Joined: Mon Apr 23, 2012 4:28 pm
Location: Travelling through time and space...

Re: SNMP Trap Not Working

Post by slansing »

Hey loopie! I just spoke with you on the phone and you mentioned you were going to send a ticket in for integration with XI. I wanted to check and see if you were planning on doing that still, if not we can continue to work through here.
loopie
Posts: 5
Joined: Wed Mar 26, 2014 5:26 pm

Re: SNMP Trap Not Working

Post by loopie »

Worked with sales to update our account, then updating to email.
thx..
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: SNMP Trap Not Working

Post by tmcdonald »

I saw that you sent in a ticket, so I will be closing this thread now. We'll deal with everything moving forward in the ticket.
Former Nagios employee
Locked