SNMP Traps Set up issue

[patrickhu]

Hi guys, I need suggestions regarding setting up the SNMP trap on Nagios XI.


I am flowing up the document “Nagios XI - Integrating SNMP Traps”, but it does not help.

I tried to send a test SNMP trap from the HP Onboard Administrator (HP Blade Enclosure C7000) to Nagios XI with the community string 0nT1m3Run or public, the CentOS received it, however the no log were showing on SNMPtrapd and SNMPTT. Nagios Trap Service showed no event.


[root@localhost snmptt]# tcpdump dst port 162
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:47:07.731557 IP 10.161.166.140.remote-as > 10.161.170.64.snmptrap: C=0nT1m3Run Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630271911 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:07.731698 IP 10.161.166.140.brvread > 10.161.170.64.snmptrap: Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630271911 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:19.168925 IP 10.161.166.140.remote-as > 10.161.170.64.snmptrap: C=0nT1m3Run Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630273056 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:19.169577 IP 10.161.166.140.brvread > 10.161.170.64.snmptrap: Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630273056 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:51.160089 IP 10.161.166.140.remote-as > 10.161.170.64.snmptrap: C=0nT1m3Run Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630276255 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:51.162815 IP 10.161.166.140.brvread > 10.161.170.64.snmptrap: Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630276255 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
[root@localhost snmptt]# tcpdump dst port 162
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:47:07.731557 IP 10.161.166.140.remote-as > 10.161.170.64.snmptrap: C=0nT1m3Run Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630271911 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:07.731698 IP 10.161.166.140.brvread > 10.161.170.64.snmptrap: Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630271911 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:19.168925 IP 10.161.166.140.remote-as > 10.161.170.64.snmptrap: C=0nT1m3Run Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630273056 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
09:47:19.169577 IP 10.161.166.140.brvread > 10.161.170.64.snmptrap: Trap(155) E:232 10.161.166.140 enterpriseSpecific s=11003 1630273056 system.sysName.0="OA-D8D3856129A3" E:232.11.2.11.1=0 E:232.11.2.8.1="HP Onboard Administrator Test Trap sent from enclosure: VVFAT"
^C
6 packets captured
967 packets received by filter
833 packets dropped by kernel
170 packets dropped by interface


Nagios Trap Service shows nothing on the host. BTW, the TRAP Service was set up from the SNMP TRAP wizard.


SNMPTT Logs:

[root@localhost snmptt]# ls -la
total 32
drwxr-xr-x 2 root root 4096 Jan 10 11:00 .
drwxr-xr-x. 8 root root 4096 Jan 17 16:13 ..
-rw-r--r-- 1 root root 0 Jan 10 11:00 snmptt.log
-rw-r--r-- 1 root root 18972 Oct 31 10:15 snmpttsystem.log
-rw-r--r-- 1 root root 0 Jan 10 11:00 snmpttunknown.log

You can see, no logs are updated at all.


SNMPTRAPD Log:

[root@localhost snmptt]# cat /var/log/net-snmpd.log
NET-SNMP version 5.5
[root@localhost snmptt]#


Here are some configuration files regarding snmptrapd:

1. Snmptrapd.conf (under /etc/snmp)

disableAuthorization yes

traphandle default /usr/sbin/snmptt

#donotlogtraps yes

authCommunity log,execute,net public
authCommunity log,execute,net 0nT1m3Run

createUser MTM MD5 tcms2009 DES tcms2009
authUser log,execute,net MTM


2. Snmptrapd (under /etc/rc.d/init.d)

#!/bin/bash

# ucd-snmp init file for snmptrapd
#
# chkconfig: - 50 50
# description: Simple Network Management Protocol (SNMP) Trap Daemon
#
# processname: /usr/sbin/snmptrapd
# config: /etc/snmp/snmptrapd.conf
# config: /usr/share/snmp/snmptrapd.conf
# pidfile: /var/run/snmptrapd.pid


### BEGIN INIT INFO
# Provides: snmptrapd
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $network
# Should-Start:
# Should-Stop:
# Default-Start:
# Default-Stop:
# Short-Description: start and stop Net-SNMP trap daemon
# Description: Simple Network Management Protocol (SNMP) trap daemon
### END INIT INFO

# source function library
. /etc/init.d/functions

OPTIONS="-Lsd -On -p /var/run/snmptrapd.pid"
if [ -e /etc/sysconfig/snmptrapd ]; then
. /etc/sysconfig/snmptrapd
fi

RETVAL=0
prog="snmptrapd"
binary=/usr/sbin/snmptrapd
pidfile=/var/run/snmptrapd.pid

start() {
[ -x $binary ] || exit 5
echo -n $"Starting $prog: "
daemon --pidfile=$pidfile /usr/sbin/snmptrapd $OPTIONS
RETVAL=$?
echo
touch /var/lock/subsys/snmptrapd
return $RETVAL
}

stop() {
echo -n $"Stopping $prog: "
killproc -On -p $pidfile /usr/sbin/snmptrapd
RETVAL=$?
echo
rm -f /var/lock/subsys/snmptrapd
return $RETVAL
}

reload(){
stop
start
}

restart(){
stop
start
}

condrestart(){
[ -e /var/lock/subsys/snmptrapd ] && restart
return 0
}

case "$1" in
start)
start
RETVAL=$?
;;
stop)
stop
RETVAL=$?
;;
restart)
restart
RETVAL=$?
;;
reload|force-reload)
reload
RETVAL=$?
;;
condrestart|try-restart)
condrestart
RETVAL=$?
;;
status)
status snmptrapd
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|reload|force-reload}"
RETVAL=2
esac

exit $RETVAL

3. 10.161.166.140.cfg (under /usr/local/ngaios/etc/services) --- HP Onboard Administrator

###############################################################################
#
# Service configuration file
#
# Created by: Nagios QL Version 3.0.3
# Date: 2013-01-16 23:53:31
# Version: Nagios 3.x config file
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios QL will overwite all manual settings during the next update
#
###############################################################################

define service {
host_name 10.161.166.140
service_description TRAP
use xiwizard_snmptrap_service
is_volatile 1
max_check_attempts 1
check_interval 1
retry_interval 1
active_checks_enabled 0
passive_checks_enabled 1
check_period xi_timeperiod_24x7
notification_interval 1
notification_period xi_timeperiod_24x7
notification_options w,c,u,
notifications_enabled 1
contacts nagiosadmin
stalking_options o,w,c,u,
icon_image snmptrap.png
_xiwizard snmp_trap
register 1
}

define service {
host_name 10.161.166.140
service_description Uptime
use xiwizard_snmp_service
check_command check_xi_service_snmp! -o sysUpTime.0 -C 0nT1m3Run -P 2c
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts nagiosadmin
_xiwizard snmp
register 1
}

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################


4. Submit_check_result script (under /usr/local/ngaios/libexec/eventhandlers permission: -rwxr-xr-x 1 root root 1182 Jan 9 11:09)




All SNMPD, SNMPTRAPD and SNMPTT services are running on Nagios.

[root@localhost ~]# ps -ef | grep snmpd
root 4371 1 0 11:14 ? 00:00:00 /usr/sbin/snmpd -LS0-6d -Lf /dev/null -p /var/run/snmpd.pid
root 4952 2936 0 11:15 pts/0 00:00:00 grep snmpd
[root@localhost ~]# ps -ef | grep snmptt
root 1458 1 0 11:11 ? 00:00:00 /usr/bin/perl /usr/local/sbin/snmptt --daemon
snmptt 1460 1458 0 11:11 ? 00:00:00 /usr/bin/perl /usr/local/sbin/snmptt --daemon
root 4980 2936 0 11:15 pts/0 00:00:00 grep snmptt
[root@localhost ~]# ps -ef | grep snmptrapd
root 1449 1 0 11:11 ? 00:00:00 /usr/sbin/snmptrapd -On -Lsd -p /var/run/snmptrapd.pid
root 5118 2936 0 11:15 pts/0 00:00:00 grep snmptrapd
You have new mail in /var/spool/mail/root
[root@localhost ~]#


Here are more info:

OS : CentOS release 6.3 (Final)
Nagios XI : 2012 R1.3
NET-SNMP : Ver 5.5 (snmpd & snmptrapd)
SNMPTT : Ver 1.3
Perl: Ver 5.17.4
SNMP community string: 0nT1m3Run


The problem is that the trap arrived at the CentOS (hence not the firewall issue), but snmptrapd does not get and log it. Hence, the snmptt could not get it too.

Can someone shed me a light? Thanks so much!!

Patrick

[scottwilkerson]

Can you cat the snmpttsystem.log to see if there are errors in there?

[patrickhu]

Thanks Scott.

Here is the last part of the snmpttsystemlog:(No update since 31/10/2012)

Tue Oct 30 11:47:34 2012 SNMPTT v1.3 shutdown
Tue Oct 30 11:47:34 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 12:03:06 2012 SNMPTT v1.3 started
Tue Oct 30 12:03:06 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 12:03:06 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 12:03:06 2012 Could not convert user id 'snmptt' to a numeric UID

Tue Oct 30 15:02:19 2012 SNMPTT v1.3 shutdown
Tue Oct 30 15:02:19 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 17:07:48 2012 SNMPTT v1.3 started
Tue Oct 30 17:07:49 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 17:07:49 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 17:07:49 2012 Could not convert user id 'snmptt' to a numeric UID

Tue Oct 30 17:13:15 2012 SNMPTT v1.3 shutdown
Tue Oct 30 17:13:15 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 17:13:47 2012 SNMPTT v1.3 started
Tue Oct 30 17:13:47 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 17:13:47 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 17:13:48 2012 Could not convert user id 'snmptt' to a numeric UID

Wed Oct 31 09:54:35 2012 SNMPTT v1.3 shutdown
Wed Oct 31 09:54:35 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Wed Oct 31 09:55:30 2012 SNMPTT v1.3 started
Wed Oct 31 09:55:30 2012 Loading /etc/snmp/snmptt.conf
Wed Oct 31 09:55:30 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Wed Oct 31 09:55:30 2012 Could not convert user id 'snmptt' to a numeric UID

Wed Oct 31 09:57:04 2012 SNMPTT v1.3 shutdown
Wed Oct 31 09:57:04 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Wed Oct 31 10:15:44 2012 SNMPTT v1.3 started
Wed Oct 31 10:15:45 2012 Loading /etc/snmp/snmptt.conf
Wed Oct 31 10:15:45 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Wed Oct 31 10:15:45 2012 Could not convert user id 'snmptt' to a numeric UID

[patrickhu]

Thanks Scott, here is the snmpttsystemlog, no update since 31/10/2012:


Tue Oct 30 11:47:34 2012 SNMPTT v1.3 shutdown
Tue Oct 30 11:47:34 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 12:03:06 2012 SNMPTT v1.3 started
Tue Oct 30 12:03:06 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 12:03:06 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 12:03:06 2012 Could not convert user id 'snmptt' to a numeric UID

Tue Oct 30 15:02:19 2012 SNMPTT v1.3 shutdown
Tue Oct 30 15:02:19 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 17:07:48 2012 SNMPTT v1.3 started
Tue Oct 30 17:07:49 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 17:07:49 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 17:07:49 2012 Could not convert user id 'snmptt' to a numeric UID

Tue Oct 30 17:13:15 2012 SNMPTT v1.3 shutdown
Tue Oct 30 17:13:15 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Tue Oct 30 17:13:47 2012 SNMPTT v1.3 started
Tue Oct 30 17:13:47 2012 Loading /etc/snmp/snmptt.conf
Tue Oct 30 17:13:47 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Tue Oct 30 17:13:48 2012 Could not convert user id 'snmptt' to a numeric UID

Wed Oct 31 09:54:35 2012 SNMPTT v1.3 shutdown
Wed Oct 31 09:54:35 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Wed Oct 31 09:55:30 2012 SNMPTT v1.3 started
Wed Oct 31 09:55:30 2012 Loading /etc/snmp/snmptt.conf
Wed Oct 31 09:55:30 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Wed Oct 31 09:55:30 2012 Could not convert user id 'snmptt' to a numeric UID

Wed Oct 31 09:57:04 2012 SNMPTT v1.3 shutdown
Wed Oct 31 09:57:04 2012 Total traps received=0,Total traps translated=0,Total traps ignored=0,Total unknown traps=0
Wed Oct 31 10:15:44 2012 SNMPTT v1.3 started
Wed Oct 31 10:15:45 2012 Loading /etc/snmp/snmptt.conf
Wed Oct 31 10:15:45 2012 Finished loading 6666 lines from /etc/snmp/snmptt.conf
Wed Oct 31 10:15:45 2012 Could not convert user id 'snmptt' to a numeric UID

[scottwilkerson]

We are likely going to have to look at the syslog to see if there is info showing up for snmptrapd

Code: Select all

cat /var/log/messages|grep snmptrapd

[patrickhu]

We are likely going to have to look at the syslog to see if there is info showing up for snmptrapd

Code: Select all

cat /var/log/messages|grep snmptrapd

Thanks Scott.
After I completely stop the Firewall on the Centos, I can see the traps coming through.
It's wired that why the TCPDUMP could get the traps, while no syslog or snmptt logs at all? (Perviously I have set up the rules to allow the SNMP trap traffic through.)
As we are monitoring a large network, may I know the precedures to set up Nagios to monitor SNMPV3 notifications? Thanks again.

[scottwilkerson]

Glad to hear you got it working.

I haven't done this but I believe you just need to add the user to your snmptrapd.conf

http://net-snmp.sourceforge.net/wiki/in ... _TRAP_User

[patrickhu]

Dear Scott,

We still have some issues with SNMP trap.

When some devices send traps to Nagios, the trap does not show up at Nagios. However, it can be found in /var/log/snmptt/snmpttunknown.log .

Question1:

If the trap has been put into the unknown log by the SNMPTT, Nagios would not show it up at the SNMP Trap Service???

I have used the following command to feed SNMPTT with the MIBs:

i.e. the UPS Mib

snmpttconvertmib –in=/usr/local/share/snmp/mibs/UPS.mib –out=/etc/snmp/snmptt.conf.ups –exec=’/usr/local/ngaios/libexec/eventhandlers/submit_check_result $r “SNMP Traps” 1’

then

add the snmptt.conf.ups into the /etc/snmp/snmptt.ini file.


Question 2:

Be aware about above Red marks SNMP Traps

According to the document, when running above command, SNMP Traps must match with the Trap Service description.

I tried to change the passive service description from SNMP Traps (this is default by using the SNMP trap wizard ) to TRAP, but Nagios would not receive and show any traps after that.

I am worrying about the “ ” for the “SNMP Traps” phrase on above command. I am not 100% sure if it works correctly even the command runs successfully.


Any suggestions?

Patrick

[scottwilkerson]

If you have the mibs you will need to add them with the following procedure from page 2 of the document
http://assets.nagios.com/downloads/nagi ... ith_XI.pdf

Installing MIBs
You may need to configure SNMPTT on the Nagios XI server to use the MIBs your remote devices are using. This may mean having to
load extra MIBs into the /usr/share/snmp/mibs/ directory on the Nagios XI server. This can be done through the command line or also
through the XI interface Admin -> Manage MIBs.

You will then also have to run the following command to import each new MIB (replace <PathToNewMIB> with the path to the MIB you
want to import):

Code: Select all

# addmib <PathToNewMIB>

[patrickhu]

Thanks for your reply, Scott.
Actually, I did. I went into the mibs directory, and run:
Addmib *
But it didn't help.

I found the following script /usr/local/bin/snmptraphandling.pl
It looks like a new script to replace the submit_check_result.
Do you have an idea for it?
Pat