Need Help Setting UP Passive SNMP Receiver

[wipeout630]

I am using RHEL 6.5 and Nagios XI 2012 R2.9. I have run the SNMP Trap Monitoring wizard, configured it for two hosts, and uploaded the MIBs. Using TCPDUMP I can see the traps are being transmitted to the server but they go no further. SNMPD and SNMPTT daemons are running, SNMPTT is running in debug but nothing is displayed in the debug logs. I'm at a complete loss here, can someone please point me in the right direction? I've followed the PDF in the link below to the letter but it is not working.

http://assets.nagios.com/downloads/nagi ... ios_XI.pdf

[slansing]

Can you attach your snmptt.log and snmpttunknown.log for reference? We should check your exec lines as well to make sure they are being sent properly:

Code: Select all

grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10

Did you add the mibs through the web interface or use addmib? We need to make sure they processed as well, that should drop new exec lines in for them.

[wipeout630]

The snmptt.log and snmpttunknown.log files do not exist. The snmptt.debug file only shows "Sleeping for 5 seconds" and snmpttsystem.log shows as follows:

Code: Select all

Sun May 25 03:43:03 2014 Reloading configuration file(s)
Sun May 25 03:43:03 2014 Loading /usr/share/snmp/mibs/processed_mibs/INFINERA-TRAP-MIB.txt
Sun May 25 03:43:03 2014 Finished loading 136 lines from /usr/share/snmp/mibs/processed_mibs/INFINERA-TRAP-MIB.txt
Sun May 25 03:43:03 2014 Loading /etc/snmp/snmptt.conf
Sun May 25 03:43:03 2014 Finished loading 64 lines from /etc/snmp/snmptt.conf
Sun May 25 03:43:03 2014 Loading /etc/snmp/snmptt.conf.
Sun May 25 03:43:03 2014 Finished loading 156 lines from /etc/snmp/snmptt.conf.

Output from the exec check:

Code: Select all

[<redacted> snmptt]$ grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"
[<redacted> snmptt]$ grep -i 'exec' /etc/snmp/snmptt.conf. | tail -n 10
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Conditions/Alarms  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Audit events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Admin events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Security events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing TCA events  $*"

In reference to the MIBs, I added one set through the addmib command but then added another set through the web interface. I also just upgraded to NagiosXI 2014R1.0 but still experiencing the problem. I can see the traps arriving when I perform a TCPDump and I see the UDP connection messages in syslog but the traps never seem to move beyond.

[sreinhardt]

Let's get some more details here. Try out the following commands and post back the results please.

Versions installed

Code: Select all

    rpm -qa | grep snmp

Looking for bins:

Code: Select all

    ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
    ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
    ls -lva /usr/sbin | grep -i 'snmp\|addmib'

snmptt\trapd settings and user perms:

Code: Select all

    grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
    grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
    grep -i 'nag' /etc/group
    grep -i 'snmp' /etc/group
    cat /etc/snmp/snmptrapd.conf 

Checking log and spool dirs:

Code: Select all

    ll /var/log/snmptt/
    ll -d /var/log/snmptt/
    ll /var/spool/snmptt | tail -n 20
    ll -d /var/spool/snmptt

Service status:

Code: Select all

    service snmptt status
    service snmptrapd status

[wipeout630]

Versions:

Code: Select all

# rpm -qa | grep snmp
net-snmp-5.5-49.el6_5.1.i686
nagios-plugins-snmp-1.4.16-10.el6.i686
net-snmp-utils-5.5-49.el6_5.1.i686
snmptt-1.4-0.9.beta2.el6.noarch
net-snmp-perl-5.5-49.el6_5.1.i686
php-snmp-5.3.3-27.el6_5.i686
net-snmp-libs-5.5-49.el6_5.1.i686

Bins:

Code: Select all

# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root nagios      804 Mar 31 12:22 addmib
-r-xr-xr-x   1 root root       4817 Oct 21  2013 snmpkey
-rwxr-xr-x   1 root root       2078 Mar 31 12:22 snmptraphandling.py
-rwxr-xr-x   1 root root      30438 Mar 31 12:22 snmpttconvertmib

# ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'

# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x   1 root root        25972 Mar  6 05:50 snmpd
-rwxr-xr-x   1 root root        25992 Mar  6 05:50 snmptrapd
-rwxr-xr-x   1 root root       177466 Oct 22  2012 snmptt
-rwxr-xr-x   1 root root         6493 Oct 22  2012 snmptthandler

SNMPtt\trapd settings:

Code: Select all

# grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt

# grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (coldStart)"
#EXEC qpage -f TRAP notifygroup1 "Device reinitialized (warmStart)"
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1.  Admin state: $2.  Operational state: $3"
#EXEC qpage -f TRAP notifygroup1 "SNMP authentication failure"

# grep -i 'nag' /etc/group
nagcmd:x:20003:apache,nagios,snmptt
nagios:x:20004:nagios,apache,snmptt

# grep -i 'snmp' /etc/group
snmptt:x:496:snmptt
nagcmd:x:20003:apache,nagios,snmptt
nagios:x:20004:nagios,apache,snmptt

# cat /etc/snmp/snmptrapd.conf
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler

Log & Spool dirs:

Code: Select all

# ll /var/log/snmptt/
total 12056
-rw-rw-r-- 1 snmptt snmptt  579888 Jun  9 12:44 snmptt.debug
-rw-rw-r-- 1 snmptt snmptt 2931588 May 18 03:22 snmptt.debug-20140518
-rw-rw-r-- 1 snmptt snmptt 2933650 May 25 03:43 snmptt.debug-20140525
-rw-rw-r-- 1 snmptt snmptt 2924320 Jun  1 03:32 snmptt.debug-20140601
-rw-rw-r-- 1 snmptt snmptt 2928446 Jun  8 03:35 snmptt.debug-20140608
-rw-rw-r-- 1 snmptt snmptt     532 Jun  8 03:35 snmpttsystem.log
-rw-rw-r-- 1 snmptt snmptt     532 May 11 03:08 snmpttsystem.log-20140518
-rw-rw-r-- 1 snmptt snmptt     532 May 18 03:22 snmpttsystem.log-20140525
-rw-rw-r-- 1 snmptt snmptt     532 May 25 03:43 snmpttsystem.log-20140601
-rw-rw-r-- 1 snmptt snmptt     532 Jun  1 03:32 snmpttsystem.log-20140608

# ll -d /var/log/snmptt/
drwxrwxr-x. 2 snmptt snmptt 4096 Jun  8 03:35 /var/log/snmptt/

# ll /var/spool/snmptt | tail -n 20
total 0

# ll -d /var/spool/snmptt
drwxrwxr-x. 2 snmptt snmptt 4096 Jan 18 11:44 /var/spool/snmptt

Services:

Code: Select all

# service snmptt status
snmptt (pid  19149) is running...

# service snmptrapd status
snmptrapd (pid  1696) is running...

[sreinhardt]

Everything there looks great! If you are not getting traps, is snmptrapd started?

Code: Select all

service snmptrapd status

When traps come in, snmptrapd takes them from the network and spool them in /var/spool/snmptt/. Snmptt then picks up the spooled traps and uses snmptt.conf and any imports for finding how to handle the traps. Specifically for XI you are interested in the EXEC lines of snmptt.conf, which it looks like you presently have not imported any so they are still default.

[wipeout630]

SNMPTrapd is running. Here is the output of my snmptt.conf file specific to the MIB I am working with:

Code: Select all

cat snmptt.conf. | grep EXEC
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Conditions/Alarms  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Audit events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Admin events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing Security events  $*"
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 1 "SNMP Notification representing TCA events  $*"

Theoretically, this should be working now but I don't see anything in the logs, spool directory, or in Nagios.

[wipeout630]

I can see that snmpd is registering incoming traps because I see the incoming UDP connection in syslog:

Code: Select all

Jun 11 09:38:09 limelight snmpd[10521]: Connection from UDP: [10.120.2.20]:300->[10.6.1.109]

[wipeout630]

I found the first problem, I had both snmpd and snmptrapd configured. Disabled snmpd and reconfigured snmptrapd to listen on port 161 and I now see data in the spool directory as well as the log files. The traps are still not being passed to Nagios but I am continuing to troubleshooting.

[sreinhardt]

Traps should never be coming in on 161, port 162 is where they should be passed. This is why snmpd runs on 161 for get requests. I would highly suggest you change it back, and correct whatever devices you have configured to send traps over 161.