Associate SNMP Trap with a Host/Service

[derekb]

Hi there,

This is has probably been asked a hundred times over, and I've read thru the following KB Article:
https://support.nagios.com/kb/article.php?id=77

How do I get a trap sent to NAGIOS XI to associate to a Host AND a service?

For example, a device on my network sent some traps to XI (snip from snmptt.log)

Code: Select all

Mon Aug 29 00:00:02 2016 .1.3.6.1.4.1.24681.1.10.0.1 Normal "Status Events" 10.1.1.50 - Info: [HDD SMART] Host: Disk 2 Rapid Test started.
Mon Aug 29 00:00:02 2016 .1.3.6.1.4.1.24681.1.10.0.1 Normal "Status Events" 10.1.1.50 - Info: [HDD SMART] Host: Disk 2 Rapid Test started.
Mon Aug 29 00:00:04 2016 .1.3.6.1.4.1.24681.1.10.0.1 Normal "Status Events" 10.1.1.50 - Info: [HDD SMART] Host: Disk 4 Rapid Test started.

If I go to Unconfigured Objects, I see the device IP and a potential service call "SNMP Traps". How would I break this down further? In most cases I have a Host monitoring multiple devices on that network which are setup as Services.

So an example:

HOST -> Site A
Service -> Device 1
Service -> Device 1 SNMP Traps

We primarily use NRDS to monitor remote hosts. Ideally what I'm looking for is to create separate SNMP TRAP services to the tune of something like "if I receive a trap from X IP, associate to service Y".

Is this possible?

[Box293]

Your would need multiple EVENTS in snmptt.conf

It's the EXEC line that targets the "SNMP Traps" service.

So you would need to use a MATCH statement in each EVENT to differentiate one from another (this allows for multiple EVENTS for the same OID).

Does that make sense?

[derekb]

Your would need multiple EVENTS in snmptt.conf

It's the EXEC line that targets the "SNMP Traps" service.

So you would need to use a MATCH statement in each EVENT to differentiate one from another (this allows for multiple EVENTS for the same OID).

Does that make sense?

I think so. So for each event I want to trigger an alert for I need to match on IP address received. This would mean if I had 100 of the same device sending traps and I wanted a separate service for each, I'd need 100 separate events/matches , correct?

[Box293]

Yes and no, it gets complicated.

You could have one EVENT for HDD SMART, instead of "SNMP Traps" your EXEC line has "HDD SMART"

In the EXEC line, "$r" is the name it uses to send it to Nagios, this is how Nagios knows which host it's for.

They key is the OID in the EVENT line. As long as .1.3.6.1.4.1.24681.1.10.0.1 was always for HDD SMART then it would be fine. But if that OID was also for something else, like HDD temperature, then you would need to have multiple EVENT's using MATCH statements to differentiate them.

Does that help?

SNMPTT can seem complicated but is quite powerful, once you get your head around it you can do a lot.

Keep asking questions if I'm not helping.

[derekb]

Helping so far!
My problem is what if I have multiple devices on different IPs reporting traps on that OID?
I need different Services for each IP/device.

Not sure how I'd set that up, or how I'd make that entry in my NRDS config

[Box293]

Yes you would need a service for each device. If they don't already exist in XI, you can use the unconfigured objects wizard to add them.

Alternatively:
Create a hostgroup that contains all the hosts that traps are coming in from (assuming they already exist)
You can then assign the SNMP service to the hostgroup
All those hosts will then have that SNMP Traps service
This solution requires adding these hosts to XI ahead of time. You can't use the wizard using this method as it defeats the purpose of using a hostgroup. I'm assuming the hosts already exist as you are using NRDS.

[derekb]

Sorry, I stepped away for a bit and am revisiting this now. I am having a hard time putting it all together... Let's use this for an example. I have multiple devices using NAS-MIB (QNAP device) that is sending traps. snmptt.conf is configured as follows:

Code: Select all

MIB: NAS-MIB (file:/usr/share/snmp/mibs/NAS.mib) converted on Wed Aug 17 15:50:21 2016 using snmpttconvertmib v1.4beta2
#
#
#
EVENT eventInform .1.3.6.1.4.1.24681.1.10.0.1 "Status Events" Normal
FORMAT Info: $1
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Info: $1"
SDESC
Info: %s
Variables:
  1: eventInformMsg
EDESC
#
#
#
EVENT eventWarning .1.3.6.1.4.1.24681.1.10.0.2 "Status Events" Normal
FORMAT $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "$*"
SDESC

Warn: %s
Variables:
  1: eventWarningMsg
EDESC
#
#
#
EVENT eventError .1.3.6.1.4.1.24681.1.10.0.4 "Status Events" Normal
FORMAT $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "$*"
SDESC

Error: %s
Variables:
  1: eventErrorMsg
EDESC

I have a small Linux device at customer premise. Using NRDS, I have setup check_ping for 2 QNAP devices:

Code: Select all

command[__HOST__]=/usr/lib/nagios/plugins/check_ping -H localhost -w 200.0,40% -c 400.0,80% -p 1
command[QNAP 1]=/usr/lib/nagios/plugins/check_ping -H 192.168.103.201 -w 200.0,40% -c 400.0,80% -p 5
command[QNAP 2]=/usr/lib/nagios/plugins/check_ping -H 192.168.103.202 -w 200.0,40% -c 400.0,80% -p 5

The host is setup in XI, and I have 2 services, QNAP1 and QNAP2. At this point, I'm deciding if I want to run snmptt and snmptrapd locally on the Linux device or just have the QNAP send traps direct to XI via WAN -- for the example lets say I'm having the QNAP send traps directly to XI.

QNAP 1 sends a trap with OID .1.3.6.1.4.1.24681.1.10.0.4 to XI. If i look in Admin > Unconfigured Objects, I'll see a new object for "SNMP Traps" with the WAN IP of the customers site. If i add that object, everything is great, however the SNMP Traps are tied to the NRDS Hosts and not either of the 2 services (QNAP1 or QNAP2). I want to make a new service such as SNMP Traps - QNAP 1.

I know I can do this by makign multiple entries in snmptt.conf for the same OID. However, how do I match based on the IP of the trap agent?
Example: QNAP 1 sends its trap to XI. XI notices the trap agent IP address ($aA) is 192.168.103.201.

In the snmptt docs, I see I can use something like this:

Code: Select all

MATCH $x: [!] x.x.x.x

..inside of an event. So I could make multiple events for each device that would be sending that specific OID, and change ....
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Info: $1"
..to something like this:

EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps - QNAP 1" "$s" "$@" "$-*" "Info: $1"

Does this make sense so far? The one thing I'm struggling with is when I look at the snmptt log files on my NAGIOS server, I cannot figure out how to "extract" any of the variables/OBJECTS for the match statement. If we look at snmpttunknown.log:

Code: Select all

[root@nagios snmptt]# cat snmpttunknown.log
Sun Sep  4 19:11:39 2016: Unknown trap (.1.3.6.1.4.1.24681.1.10.0.1) received from 10.1.1.50 at:
Value 0: 10.1.1.50
Value 1: 10.1.1.50
Value 2: 7:19:50:01.73
Value 3: .1.3.6.1.4.1.24681.1.10.0.1
Value 4: 10.1.1.50
Value 5: empire
Value 6: .1.3.6.1.4.1.24681.1.10
Value 7:
Value 8:
Value 9:
Value 10:
Ent Value 0: .1.3.6.1.4.1.24681.1.1.101.0=[HDD SMART] Host: Disk 2 Rapid Test started.

..we can see that the IP address is not passed in any of the "Objects" -- so $1 would not give me anything useful for my MATCH statement. Am I able to reference Value 0 ~ Value 10 at all somehow? Or should I just try to match on an snmptt variable, like $aA?

From snmptt docs:

Code: Select all

The following command formats are available:

MATCH MODE=[or | and]
MATCH $x: [!] (reg) [i]
MATCH $x: [!] n
MATCH $x: [!] n-n
MATCH $x: [!] < n
MATCH $x: [!] > n
MATCH $x: [!] & n
MATCH $x: [!] x.x.x.x
MATCH $x: [!] x.x.x.x-x.x.x.x
MATCH $x: [!] x.x.x.x/x

where:
or or and set the default evaluation mode for ALL matches
$x is any variable (example: $3, $A etc)
reg is a regular expression
! is used to negate the result (not)
& is used to perform a bitwise AND
n is a number
x.x.x.x is an IP address
x.x.x.x-x.x.x.x is an IP network address range
x.x.x.x/x is an IP CIDR network addresss

I'm wondering if I am better off getting something like Fusion, and running Nagios Core at customer premise...

[Box293]

Code: Select all

[root@nagios snmptt]# cat snmpttunknown.log
Sun Sep  4 19:11:39 2016: Unknown trap (.1.3.6.1.4.1.24681.1.10.0.1) received from 10.1.1.50 at:
Value 0: 10.1.1.50
Value 1: 10.1.1.50
Value 2: 7:19:50:01.73
Value 3: .1.3.6.1.4.1.24681.1.10.0.1
Value 4: 10.1.1.50
Value 5: empire
Value 6: .1.3.6.1.4.1.24681.1.10
Value 7:
Value 8:
Value 9:
Value 10:
Ent Value 0: .1.3.6.1.4.1.24681.1.1.101.0=[HDD SMART] Host: Disk 2 Rapid Test started.

Value 0 - 10 fields are all the information about the sending server, the device that actually sends the trap. These are accessed like $a, $C etc.
Ent Value x fields are the $1, $2 etc objects.

QNAP 1 sends a trap with OID .1.3.6.1.4.1.24681.1.10.0.4 to XI. If i look in Admin > Unconfigured Objects, I'll see a new object for "SNMP Traps" with the WAN IP of the customers site. If i add that object, everything is great, however the SNMP Traps are tied to the NRDS Hosts and not either of the 2 services (QNAP1 or QNAP2). I want to make a new service such as SNMP Traps - QNAP 1.

I know I can do this by makign multiple entries in snmptt.conf for the same OID. However, how do I match based on the IP of the trap agent?
Example: QNAP 1 sends its trap to XI. XI notices the trap agent IP address ($aA) is 192.168.103.201.

..we can see that the IP address is not passed in any of the "Objects" -- so $1 would not give me anything useful for my MATCH statement.

Correct, because the address isn't coming in then it's going to be hard to match against it. If there is something specific in the incoming trap then you could match it. For example traps by default in XI don't use any authentication. So perhaps you could set the sending SNMP Community string on the device to be the IP address of the device, then you could use $C instead of $r in the EXEC line.

Does that help?

[derekb]

Yep! That gives me enough to go on for now!

[Box293]

Great, let us know how it goes.