Page 1 of 1
Need Help Setting UP Passive SNMP Receiver
Posted: Mon Apr 21, 2014 5:37 pm
by wipeout630
I am using RHEL 6.5 and Nagios XI 2012 R2.9. I have run the SNMP Trap Monitoring wizard, configured it for two hosts, and uploaded the MIBs. Using TCPDUMP I can see the traps are being transmitted to the server but they go no further. SNMPD and SNMPTT daemons are running, SNMPTT is running in debug but nothing is displayed in the debug logs. I'm at a complete loss here, can someone please point me in the right direction? I've followed the PDF in the link below to the letter but it is not working.
http://assets.nagios.com/downloads/nagi ... ios_XI.pdf
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Tue Apr 22, 2014 3:24 pm
by slansing
Can you attach your snmptt.log and snmpttunknown.log for reference? We should check your exec lines as well to make sure they are being sent properly:
Code: Select all
grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
Did you add the mibs through the web interface or use addmib? We need to make sure they processed as well, that should drop new exec lines in for them.
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Thu May 29, 2014 12:37 pm
by wipeout630
The snmptt.log and snmpttunknown.log files do not exist. The snmptt.debug file only shows "Sleeping for 5 seconds" and snmpttsystem.log shows as follows:
Code: Select all
Sun May 25 03:43:03 2014 Reloading configuration file(s)
Sun May 25 03:43:03 2014 Loading /usr/share/snmp/mibs/processed_mibs/INFINERA-TRAP-MIB.txt
Sun May 25 03:43:03 2014 Finished loading 136 lines from /usr/share/snmp/mibs/processed_mibs/INFINERA-TRAP-MIB.txt
Sun May 25 03:43:03 2014 Loading /etc/snmp/snmptt.conf
Sun May 25 03:43:03 2014 Finished loading 64 lines from /etc/snmp/snmptt.conf
Sun May 25 03:43:03 2014 Loading /etc/snmp/snmptt.conf.
Sun May 25 03:43:03 2014 Finished loading 156 lines from /etc/snmp/snmptt.conf.
Output from the exec check:
Code: Select all
[<redacted> snmptt]$ grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1. Admin state: $2. Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1. Admin state: $2. Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"
[<redacted> snmptt]$ grep -i 'exec' /etc/snmp/snmptt.conf. | tail -n 10
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Conditions/Alarms $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Audit events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Admin events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Security events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing TCA events $*"
In reference to the MIBs, I added one set through the addmib command but then added another set through the web interface. I also just upgraded to NagiosXI 2014R1.0 but still experiencing the problem. I can see the traps arriving when I perform a TCPDump and I see the UDP connection messages in syslog but the traps never seem to move beyond.
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Thu May 29, 2014 2:18 pm
by sreinhardt
Let's get some more details here. Try out the following commands and post back the results please.
Versions installed
Looking for bins:
Code: Select all
ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
ls -lva /usr/sbin | grep -i 'snmp\|addmib'
snmptt\trapd settings and user perms:
Code: Select all
grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
grep -i 'nag' /etc/group
grep -i 'snmp' /etc/group
cat /etc/snmp/snmptrapd.conf
Checking log and spool dirs:
Code: Select all
ll /var/log/snmptt/
ll -d /var/log/snmptt/
ll /var/spool/snmptt | tail -n 20
ll -d /var/spool/snmptt
Service status:
Code: Select all
service snmptt status
service snmptrapd status
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Mon Jun 09, 2014 2:47 pm
by wipeout630
Versions:
Code: Select all
# rpm -qa | grep snmp
net-snmp-5.5-49.el6_5.1.i686
nagios-plugins-snmp-1.4.16-10.el6.i686
net-snmp-utils-5.5-49.el6_5.1.i686
snmptt-1.4-0.9.beta2.el6.noarch
net-snmp-perl-5.5-49.el6_5.1.i686
php-snmp-5.3.3-27.el6_5.i686
net-snmp-libs-5.5-49.el6_5.1.i686
Bins:
Code: Select all
# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
-rwxr-xr-x 1 root nagios 804 Mar 31 12:22 addmib
-r-xr-xr-x 1 root root 4817 Oct 21 2013 snmpkey
-rwxr-xr-x 1 root root 2078 Mar 31 12:22 snmptraphandling.py
-rwxr-xr-x 1 root root 30438 Mar 31 12:22 snmpttconvertmib
# ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x 1 root root 25972 Mar 6 05:50 snmpd
-rwxr-xr-x 1 root root 25992 Mar 6 05:50 snmptrapd
-rwxr-xr-x 1 root root 177466 Oct 22 2012 snmptt
-rwxr-xr-x 1 root root 6493 Oct 22 2012 snmptthandler
SNMPtt\trapd settings:
Code: Select all
# grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt
# grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1. Admin state: $2. Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1. Admin state: $2. Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"
# grep -i 'nag' /etc/group
nagcmd:x:20003:apache,nagios,snmptt
nagios:x:20004:nagios,apache,snmptt
# grep -i 'snmp' /etc/group
snmptt:x:496:snmptt
nagcmd:x:20003:apache,nagios,snmptt
nagios:x:20004:nagios,apache,snmptt
# cat /etc/snmp/snmptrapd.conf
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler
Log & Spool dirs:
Code: Select all
# ll /var/log/snmptt/
total 12056
-rw-rw-r-- 1 snmptt snmptt 579888 Jun 9 12:44 snmptt.debug
-rw-rw-r-- 1 snmptt snmptt 2931588 May 18 03:22 snmptt.debug-20140518
-rw-rw-r-- 1 snmptt snmptt 2933650 May 25 03:43 snmptt.debug-20140525
-rw-rw-r-- 1 snmptt snmptt 2924320 Jun 1 03:32 snmptt.debug-20140601
-rw-rw-r-- 1 snmptt snmptt 2928446 Jun 8 03:35 snmptt.debug-20140608
-rw-rw-r-- 1 snmptt snmptt 532 Jun 8 03:35 snmpttsystem.log
-rw-rw-r-- 1 snmptt snmptt 532 May 11 03:08 snmpttsystem.log-20140518
-rw-rw-r-- 1 snmptt snmptt 532 May 18 03:22 snmpttsystem.log-20140525
-rw-rw-r-- 1 snmptt snmptt 532 May 25 03:43 snmpttsystem.log-20140601
-rw-rw-r-- 1 snmptt snmptt 532 Jun 1 03:32 snmpttsystem.log-20140608
# ll -d /var/log/snmptt/
drwxrwxr-x. 2 snmptt snmptt 4096 Jun 8 03:35 /var/log/snmptt/
# ll /var/spool/snmptt | tail -n 20
total 0
# ll -d /var/spool/snmptt
drwxrwxr-x. 2 snmptt snmptt 4096 Jan 18 11:44 /var/spool/snmptt
Services:
Code: Select all
# service snmptt status
snmptt (pid 19149) is running...
# service snmptrapd status
snmptrapd (pid 1696) is running...
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Tue Jun 10, 2014 4:09 pm
by sreinhardt
Everything there looks great! If you are not getting traps, is snmptrapd started?
When traps come in, snmptrapd takes them from the network and spool them in /var/spool/snmptt/. Snmptt then picks up the spooled traps and uses snmptt.conf and any imports for finding how to handle the traps. Specifically for XI you are interested in the EXEC lines of snmptt.conf, which it looks like you presently have not imported any so they are still default.
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Tue Jun 10, 2014 5:57 pm
by wipeout630
SNMPTrapd is running. Here is the output of my snmptt.conf file specific to the MIB I am working with:
Code: Select all
cat snmptt.conf. | grep EXEC
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Conditions/Alarms $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Audit events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Admin events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Security events $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing TCA events $*"
Theoretically, this should be working now but I don't see anything in the logs, spool directory, or in Nagios.
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Wed Jun 11, 2014 11:44 am
by wipeout630
I can see that snmpd is registering incoming traps because I see the incoming UDP connection in syslog:
Code: Select all
Jun 11 09:38:09 limelight snmpd[10521]: Connection from UDP: [10.120.2.20]:300->[10.6.1.109]
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Wed Jun 11, 2014 12:44 pm
by wipeout630
I found the first problem, I had both snmpd and snmptrapd configured. Disabled snmpd and reconfigured snmptrapd to listen on port 161 and I now see data in the spool directory as well as the log files. The traps are still not being passed to Nagios but I am continuing to troubleshooting.
Re: Need Help Setting UP Passive SNMP Receiver
Posted: Thu Jun 12, 2014 2:06 pm
by sreinhardt
Traps should never be coming in on 161, port 162 is where they should be passed. This is why snmpd runs on 161 for get requests. I would highly suggest you change it back, and correct whatever devices you have configured to send traps over 161.