No file called /usr/local/nagios/etc/objects/switch.cfg

[sreinhardt]

I'd suggest looking at those debug files again. I would imagine the linkup\linkdown traps are there by default and do not have an exec line. Either that, or they are still unknown which do not get passed to xi. Either way, we should see them logged in either the debug files or the standard log files.

[perric]

Hello,

I did the debug and just see the sleep log messages in the snmptt.debug file.

cwd: /var/run
Changing to UID: snmptt (495)
Closing debug file /var/log/snmptt/snmptt.debug
Debug file /var/log/snmptt/snmptt.debug re-opened under uid 495
Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds


Yet, my Cisco router shows attempts to send SNMP to my Nagios server...

Nov 11 17:05:12 AST: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
Nov 11 17:05:12: SNMP: Queuing packet to 192.168.180.121
Nov 11 17:05:12: SNMP: V1 Trap, ent snmpTraps, addr 192.168.183.11, gentrap 2, spectrap 0
ifIndex.3 = 3
ifDescr.3 = GigabitEthernet0/1
ifType.3 = 6
lifEntry.20.3 = administratively down

David

[perric]

We rebooted our Nagios server. After the reboot, I found no new entries in snmpttunknown.log, but did find some entries in the snmptt.debug file, but then they stopped. Here are the last 30 lines or so of snmptt.debug. See below:

EXEC line(s):
OID found in cache: '.1.3.6.1.4.1.9.9.412.1.1.2.0' -> 'enterprises.9.9.412.1.1.2.0'
OID found in cache: '.1.3.6.1.4.1.9.9.412.1.1.1.0' -> 'enterprises.9.9.412.1.1.1.0'
OID found in cache: '.1.3.6.1.4.1.9.2.1.5.0' -> 'enterprises.9.2.1.5.0'

Variable .1.3.6.1.4.1.9.9.412.1.1.2.0 with value 10.100.225.43
Value appears to contain an OID or IP address.
OID found in cache: '.1.3.6.1.4.1.9.9.412.1.1.2.0' -> 'enterprises.9.9.412.1.1.2.0'

Variable .1.3.6.1.4.1.9.9.412.1.1.1.0 with value 1
Value does not appear to contain an OID
Value is numerical
Value is NOT defined as an INTEGER or Integer32 in the mib
OID found in cache: '.1.3.6.1.4.1.9.9.412.1.1.1.0' -> 'enterprises.9.9.412.1.1.1.0'

Variable .1.3.6.1.4.1.9.2.1.5.0 with value 10.100.225.43
Value appears to contain an OID or IP address.
OID found in cache: '.1.3.6.1.4.1.9.2.1.5.0' -> 'enterprises.9.2.1.5.0'

OID of received trap: .1.3.6.1.6.3.1.1.5.5. Will attempt to translate to text
OID found in cache: '.1.3.6.1.6.3.1.1.5.5' -> 'authenticationFailure'
Translated to authenticationFailure
EXEC command:/usr/local/bin/snmptraphandling.py "192.168.180.11" "SNMP Traps" "Normal" "1415809094" "enterprises.9.2.1.5.0 ():10.100.225.43 enterprises.9.9.412.1.1.1.0 ():1 enterprises.9.9.412.1.1.2.0 ():10.100.225.43" "An authenticationFailure trap signifies that the SNMP 10.100.225.43 1 10.100.225.43"

Here is output of ps aux for snmp related daemons...

root 2753 0.0 0.0 195796 3772 ? Ss 11:13 0:00 /usr/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
root 2762 0.0 0.1 165040 11780 ? Ss 11:13 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt 2763 0.0 0.1 169324 13064 ? Ss 11:13 0:00 /usr/bin/perl /usr/sbin/snmptt --daemon
root 5149 0.0 0.0 117260 3676 ? S 11:18 0:00 python /usr/local/bin/snmptraphandling.py

David

[perric]

There seems to be something wrong with my snmptt and snmptrapd daemons. When I restart them, I see traps coming in and all appears to be working, but then they stop.

I believe I am at the last step or two here. I see traps coming into Nagios, but I am not getting notifications. See below. When I view the service, it shows "Last Notification" as never...

Service State:Ok
Duration:45m 45s
State Type:Hard
Current Check:1 of 1
Last Check:2014-11-12 17:18:12
Next Check:Not scheduled
Last State Change:2014-11-12 16:34:17
Last Notification:Never
Check Type:Passive
Check Latency:4.73531 seconds
Execution Time:0 seconds
State Change:0%

I have a service template configured to notify and this template is assigned to a service (which is in turn assigned to a host). Is there any logic that would prevent a service notification? My SNMP test is by shutting down an unused Loopback interface on a router, so it should not make the host appear as down. Or is there a way to validate the notification configuration? I have a contactgroup specified that is working for host up/down and I have every alert option specified.

David

[Box293]

Are you seeing the traps appear in /usr/local/nagios/var/nagios.log as a passive service not found?

Example:

Code: Select all

Command:
tail -f /usr/local/nagios/var/nagios.log | grep 'SNMP Traps'

Output:
[1410904640] Warning:  Passive check result was received for service 'SNMP Traps' on host '192.168.207.139', but the service could not be found!

[perric]

Nothing from the tail command. Here are the last few lines from this command... cat /usr/local/nagios/var/nagios.log | grep 'SNMP Traps'. The first two entries are for the device where I am trying to receive traps.

[1415883062] SERVICE ALERT: NET_drsanrtr02-dmvpn;SNMP Traps;OK;HARD;1;An ospfOriginateLsa trap signifies that a new 192.168.183.11 0.0.0.0 5 10.201.72.254 192.168.183.11 / mib-2.14.1.1 ():192.168.183.11 mib-2.14.4.1.1 ():0.0.0.0 mib-2.14.4.1.2 ():5 mib-2.14.4.1.3 ():10.201.72.254 mib-2.14.4.1.4 ():192.168.183.11
[1415883282] SERVICE ALERT: NET_drsanrtr02-dmvpn;SNMP Traps;OK;HARD;1;An authenticationFailure trap signifies that the SNMP 10.100.225.43 1 10.100.225.43 / enterprises.9.2.1.5.0 ():10.100.225.43 enterprises.9.9.412.1.1.1.0 ():1 enterprises.9.9.412.1.1.2.0 ():10.100.225.43
[1415885499] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!
[1415885504] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!
[1415885514] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!
[1415885519] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!
[1415885529] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!
[1415885534] Warning: Passive check result was received for service 'SNMP Traps' on host '192.168.180.52', but the service could not be found!

David

[sreinhardt]

OK, so that tells me a couple things.

1) The unknown host or service note, means that it is coming in with a different ip or hostname than the host is configured with, or there is not snmp traps service for that host. Thankfully this is easy to correct in admin->unconfigured objects, and it will show all of the passive checks that have been received and do not have appropriate hosts or services for them. The unconfigured objects menu can also let you see when a host is sending as a slightly different name than you have configured or sending as an IP with no hostname at all.

2) The ones that are coming in properly are set to OK only, hence why you are not getting a notification. Traps require quite a bit more work than most other monitoring forms, as snmptt and addmib don't natively understand the idea of states. This means that once you have translated a mib or run the addmib command on it, depending on the version you have, you will either need to modify /etc/snmp/snmptt.conf or /usr/share/snmp/mibs/processed_mibs/[mibname].txt. Within these files, we are going to look at the EVENT and EXEC lines, they look something like:

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Normal
FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the SNMP entity, acting in $*"

In particular, the two highlighted portions are what dictate the status that your trap will have when set to nagios. By default all traps have an OK state, as it cannot necessarily be derived from the mib. What we can do, is change Normal, to Warning or Critical, and snmptraphandling.py has a set of keywords it will understand so that it can set the status appropriately.

A second option once you try that key word, is some traps have a Variables: line and numbered entries below that. You can replace the $s with $4 for the 4th variable and modify snmptraphandling.py to look for the additional words that variable may use to indicate state. You do not have to specifically use the 4th variable.

[perric]

I have set the trap details to Warning (as shown below) and I also tried Critical. I have restarted the snmptt and snmptrapd daemon's, but it is still not notifying. You specified a variable to change, but I am not sure what it should be. Does the python script assume that "Operational" is the first variable ($1) being passed to it?

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Warning
FORMAT Link down on interface $1. Admin state: $2. Operational state: $3
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1. Admin state: $2. Operational state: $3"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state). This other state is indicated by the included value
of ifOperStatus.
EDESC

Within the Python script is the following. Are the words case sensitve?
def get_return_code(severity):
severity = severity.upper()
if severity == "INFORMATIONAL":
return_code = "0"
elif severity == "NORMAL":
return_code = "0"
elif severity == "SEVERE":
return_code = "2"
elif severity == "MAJOR":
return_code = "2"
elif severity == "CRITICAL":
return_code = "2"
elif severity == "WARNING":
return_code = "1"
elif severity == "MINOR":
return_code = "1"
else:
printusage()
return return_code

David

[sreinhardt]

The issue here, is that you modified the link up\down trap that does not have a proper exec line. You need to look for the other entry that looks identical to my last post. Change the key word in that particular one, and restart your daemons. Since we did see the traps in the nagios log and status details, you should have a second entry. If you can't find it, please post snmptt.conf and snmptt.ini again.

[perric]

Ok. Didn't realize there were two entries and I also see that one entry had the comment character in front of the EXEC line. I found the other entry and changed it to Critical as shown below.

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Critical
FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the SNMP entity, acting in $*"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state). This other state is indicated by the included value
of ifOperStatus.
Variables:
1: ifIndex
2: ifAdminStatus
3: ifOperStatus
EDESC

But still no change on the event notification - Last Notification reads "Never". I cleared my snmptt.debug and restarted the snmptt and snmptrapd daemons. When I recreated the trap, my snmptt.debug log shows this:

Event: .1.3.6.1.6.3.1.1.5.3 => linkDown,Status Events,Critical,Link down on interface $1. Admin state: $2. Operational state: $3 ,
Event: .1.3.6.1.6.3.1.1.5.3 => linkDown,Status Events,Critical,A linkDown trap signifies that the SNMP entity, acting in $*,

I also attached my snmptt.ini. Should I be changing the $ entries to something in the EXEC line?

David