Nagios Support Forum

Thanks. Here are the results:

[root@caw snmptt]# ls -lva /usr/local/bin | grep -i 'snmp\|addmib'
-rwx--x--- 1 root nagios 804 Nov 6 11:49 addmib
-rwx------ 1 root root 2078 Nov 6 11:49 snmptraphandling.py
-rwx------ 1 root root 30438 Nov 6 11:49 snmpttconvertmib
[root@caw snmptt]# ls -lva /usr/local/sbin | grep -i 'snmp\|addmib'
[root@caw snmptt]# ls -lva /usr/sbin | grep -i 'snmp\|addmib'
-rwxr-xr-x 1 root root 30744 Oct 21 04:37 snmpd
-rwxr-xr-x 1 root root 30776 Oct 21 04:37 snmptrapd
-rwxr-xr-x 1 root root 177466 Oct 22 2012 snmptt
-rwxr-xr-x 1 root root 6493 Oct 22 2012 snmptthandler
[root@caw snmptt]# cat /etc/snmp/snmptrapd.conf | tail -n 50
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler


[root@caw snmptt]# grep -i 'daemon_uid\|mode =' /etc/snmp/snmptt.ini
mode = daemon
description_mode = 0
# A second (child) process will be started as the daemon_uid user so
daemon_uid = snmptt
[root@caw snmptt]# grep -i 'exec' /etc/snmp/snmptt.conf | tail -n 10
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An ospfVirtNbrRestartHelperStatusChange trap signifies $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A coldStart trap signifies that the SNMPv2 $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A warmStart trap signifies that the SNMPv2 $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An authenticationFailure trap signifies that the $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An egpNeighborLoss trap signifies that an EGP $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A coldStart trap signifies that the sending $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the sending $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkUp trap signifies that the sending $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An authenticationFailure trap signifies that $*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "An egpNeighborLoss trap signifies that an EGP $*"
[root@caw snmptt]# grep -i 'nag' /etc/group
nagios501:nagios,apache,snmptt
nagcmd502:nagios,apache,snmptt
[root@caw snmptt]# grep -i 'snmp' /etc/group
nagios501:nagios,apache,snmptt
nagcmd502:nagios,apache,snmptt
snmptt303:


[root@caw snmptt]# ll /var/log/snmptt/
total 128
-rw-rw-r-- 1 snmptt root 28966 Nov 7 14:40 snmptt.log
-rw-r--r-- 1 root root 238 Nov 6 11:49 snmpttsystem.log
-rw-rw-r-- 1 snmptt root 85217 Nov 7 14:08 snmpttunknown.log
[root@caw snmptt]# ll -d /var/log/snmptt/
drwxrwxr-x 2 snmptt snmptt 4096 Nov 6 16:11 /var/log/snmptt/
[root@caw snmptt]# ll /var/spool/snmptt
total 0
[root@caw snmptt]# ll -d /var/spool/snmptt
drwxrwxr-x 2 snmptt snmptt 4096 Nov 7 14:40 /var/spool/snmptt

David

Well... you have everything just about perfect from what that shows! Everything has exec lines, the permissions are great, and all the scripts and bins we need are there. Let's go ahead and turn on debugging for snmptt and see what that turns up.

Change to these values in snmptt.ini and restart the daemon: Send a few traps in, and post the /var/log/snmptt/snmptt.debug and /var/log/snmptt/snmptthandler.debug please.

Attached are the debug logs.

David

I see some valid ospf traps coming in, that appear to have been processed correctly and should either be under the 192.168.83.11 snmp traps service, or in unconfigured objects. The exec line appears to have gone great for this among a few other traps. I can see in the handler debug log, you also have a fair amount of unknown traps, but those are mostly from uptime traps, and how you would like to handle that is up to you. Do you have an snmp trap service for the host 192.168.83.11? If not, do you see some snmp trap services waiting to be applied in Admin->Unconfigured objects?

Thanks for your reply. I have no other way to describe this, except that it is suddenly working today vs last Friday. I see the trap messages now, but I have done no changes. Could there be a delay with changes taking effect?

I have one remaining problem with the service, though... it is not sending notifcation e-mails with the TRAP details. Under Last Notification, it says "Never". I do have notification working for the "host" of this service.

I have the following configured for this service via the GUI:

Config Name - NET_snmptrap_service
Description - SNMP Traps
Display Name - NET_snmptrap_service
Managed Hosts - MyHost
Template - xiwizard_snmptrap_service
Service - Active

Check Period - xi_timeperiod_24x7

Notifcation - Enable
Notification Period - xi_timeperiod_24x7
Notification Options - w, c, u , r, f, s
Notification Interval - 60
First notification delay - 1
Contact Group - same one used by host notification (which is working)

Am I missing something? How I can verify this on the OS level?

David

We might have a bug with notification delays, that was discovered last week. I'd suggest removing the notification delay from your config, and giving nagios a restart. I would expect an immediate email when there is an issue, and then one every 60 minutes beyond that. As for a delay in these settings taking effect, not that I can think of, aside from the service restarts to enable debugging. Otherwise those configs should be read on every snmptt and snmptrapd service restart and pick up your changes.

Thanks Spenser. I just learned that the server was rebooted on Sunday, so maybe that's what helped with the Traps.

I will try restarting Nagios.

David

One question about restarting Nagios. If I understand the "Apply Configuration" button, it is restarting Nagios. Do I need to do more than that for the restart?

David

Yes, Apply Configuration does restart nagios.

You can also restart Nagios by:
Admins Tab
Under XI System Component Status
Click the gear icon for Monitoring Engine
Click Restart

Ok. I did a restart as specified, but no change. When I check the Service, I see the status details below, which indicates no state change. I am assuming that this means there are no traps received. We have no firewall or packet filtering between the router and the Nagios server.

Current Status: OK (for 4d 21h 58m 56s)
Status Information: An ospfOriginateLsa trap signifies that a new 192.168.183.11 0.0.0.0 1 192.168.183.11 192.168.183.11 / mib-2.14.1.1 ():192.168.183.11 mib-2.14.4.1.1 ():0.0.0.0 mib-2.14.4.1.2 ():1 mib-2.14.4.1.3 ():192.168.183.11 mib-2.14.4.1.4 ():192.168.183.11
Performance Data:
Current Attempt: 1/1 (HARD state)
Last Check Time: 11-11-2014 09:56:59
Check Type: PASSIVE
Check Latency / Duration: N/A / 0.000 seconds
Next Scheduled Check: N/A
Last State Change: 11-06-2014 11:58:32
Last Notification: N/A (notification 0)
Is This Service Flapping? N/A
In Scheduled Downtime? NO
Last Update: 11-11-2014 09:57:26 ( 0d 0h 0m 2s ago)

Here are logs from my Cisco router showing traps being sent out...

Nov 11 10:58:03 AST: %LINK-5-CHANGED: Interface GigabitEthernet0/1, changed state to administratively down
Nov 11 10:58:04 AST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
Nov 11 10:58:04: SNMP: Queuing packet to 192.168.80.11.121
Nov 11 10:58:04: SNMP: V1 Trap, ent snmpTraps, addr 192.168.83.11, gentrap 2, spectrap 0
ifIndex.3 = 3
ifDescr.3 = GigabitEthernet0/1
ifType.3 = 6
lifEntry.20.3 = administratively down
Nov 11 10:58:04: SNMP: Queuing packet to 10.100.225.200
Nov 11 10:58:04: SNMP: V1 Trap, ent snmpTraps, addr 192.168.83.11, gentrap 2, spectrap 0
ifIndex.3 = 3
ifDescr.3 = GigabitEthernet0/1
ifType.3 = 6
lifEntry.20.3 = administratively down
Nov 11 10:58:04: SNMP: Packet sent via UDP to 192.168.80.121

Could there be something on the Nagios server blocking this?

David