Page 7 of 8
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 20, 2020 9:10 am
by emartine
Here is what I found. There are some level 1s that are considered critical from the 143 server because they exist on line 5 instead of line 6. As far as I can tell the critical alerts do show up on line 6 and that is fine but the OK seems to alternate between lines 5 and 6 which is causing false notifications. What do you suggest be done? I care more about the outcome for results of the 143 server than the 144 server since 144 is test.
See below for 3 examples.
Wed Oct 14 17:10:38 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" 143 - Received trap "oplGenericV2Trap" with variables
1"enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:1
3enterprises.4184.2.2.2.1.1.12.73.67.79.80.50.52.49.53.72.48.65.72:ICOP2415H0AH
4enterprises.4184.2.5.1.0:SR03
5enterprises.4184.2.5.2.0:1
6enterprises.4184.2.5.3.0:INIT - Server process loading definitions.
7enterprises.4184.2.5.4.0:2020-10-14 17:10:38
8enterprises.4184.2.5.8.0:7929857
9enterprises.4184.2.5.9.0:7929857"
Tue Oct 13 07:01:42 2020 .1.3.6.1.4.1.4184.2.0.2 Normal "Status Events" 143 - Received trap "oplGenericV2Trap_Ok" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:1
3enterprises.4184.2.2.2.1.1.12.73.67.79.80.50.52.49.53.72.48.65.72:ICOP2415H0AH
4enterprises.4184.2.3.2.1.1.12.73.67.79.80.50.52.49.53.72.48.65.72.7.51.77.95.83.70.51.51:3M_SF33
5enterprises.4184.2.5.1.0:IN08
6enterprises.4184.2.5.2.0:1
7enterprises.4184.2.5.3.0:ACTIVE and recovered from down state.
8enterprises.4184.2.5.4.0:2020-10-13 07:01:42
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"
Tue Oct 6 16:43:36 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" 143 - Received trap "oplGenericV2Trap" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:1
3enterprises.4184.2.2.2.1.1.12.73.67.79.80.50.52.49.53.72.48.65.72:ICOP2415H0AH
4enterprises.4184.2.3.2.1.1.12.73.67.79.80.50.52.49.53.72.48.65.72.7.76.65.66.95.51.77.50:LAB_3M2
5enterprises.4184.2.5.1.0:IN13
6enterprises.4184.2.5.2.0:4
7enterprises.4184.2.5.3.0:DOWN, Interface is not operational- ERROR status for Connection.
8enterprises.4184.2.5.4.0:2020-10-06 16:43:36 enterprises.4184.2.5.8.0:0
9enterprises.4184.2.5.9.0:0"
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 20, 2020 12:24 pm
by tgriep
One thing to try is to add another MATCH statement for when the device sends the data in the 5th field.
It seems that the 4th field has this string in it.
So try matching on 4th field for for that string and a 1 on field 5.
You will also need to set the Match Mode to AND.
Try this
Code: Select all
MATCH MODE=and
MATCH $4: (SR03)
MATCH $5: 1
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 20, 2020 1:36 pm
by emartine
So for my OK statement I should change it from
MATCH $6: 1
to
MATCH MODE=and
MATCH $4: (SR03)
MATCH $5: 1
MATCH $6: 1
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 20, 2020 4:40 pm
by tgriep
Actually, you will have to create a New Trap entry, one with what I posted Earlier and the other with the Match 6 still but I don't see what could be added to stop that one from triggering the invalid trap.
Here is the list for that trap's variables. See if you can use one of them.
Code: Select all
1: oplServiceName
2: oplServiceState
3: oplDesignName
4: oplInterfaceName
5: oplConnectionName
6: oplAlertName
7: oplAlertSeverity
8: oplAlertDescription
9: oplAlertDescription2
10: oplAlertTime
11: oplAlertExtendedSeverity
12: oplAlertMachineName
13: oplAlertEntityType
14: oplAlertPrimaryStatus
15: oplAlertSecondaryStatus
16: oplAlertURL
17: oplAlertURL2
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 27, 2020 2:15 pm
by emartine
So with undefined traps they come in like this. How would I use the variable in the trap definition?
Fri Sep 25 11:38:55 2020: Unknown trap (.1.3.6.1.4.1.4184.2.0.2) received from serverip144 at:
Value 0: serverip144
Value 1: serverip144
Value 2: 4:0:56:52.98
Value 3: .1.3.6.1.4.1.4184.2.0.2
Value 4: serverip144
Value 5: openlink
Value 6: .1.3.6.1.4.1.4184.2
Value 7:
Value 8:
Value 9:
Value 10:
Ent Value 0: .1.3.6.1.4.1.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53=Cerner OPENLink 24.1-05
Ent Value 1: .1.3.6.1.4.1.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53=1
Ent Value 2: .1.3.6.1.4.1.4184.2.2.2.1.1.13.73.67.79.84.50.52.49.53.72.48.65.72.70=ICOT2415H0AHF
Ent Value 3: .1.3.6.1.4.1.4184.2.3.2.1.1.13.73.67.79.84.50.52.49.53.72.48.65.72.70.12.80.82.69.67.95.69.68.77.95.88.77.76=PREC_EDM_XML
Ent Value 4: .1.3.6.1.4.1.4184.2.5.1.0=IN07
Ent Value 5: .1.3.6.1.4.1.4184.2.5.2.0=1
Ent Value 6: .1.3.6.1.4.1.4184.2.5.3.0=ACTIVE and the Alert process is monitoring this interface.
Ent Value 7: .1.3.6.1.4.1.4184.2.5.4.0=2020-09-25 11:38:55
Ent Value 8: .1.3.6.1.4.1.4184.2.5.8.0=0
Ent Value 9: .1.3.6.1.4.1.4184.2.5.9.0=0
Re: SNMP Trap Monitoring in Nagios XI
Posted: Tue Oct 27, 2020 2:38 pm
by tgriep
What I do is if I receive an unknown trap, I search the internet for the OID "1.3.6.1.4.1.4184.2.0.2" to find the MIB file.
Once I find the MIB file, I upload it in XI using the Admin > Manage MIBs menu so it will create the Trap Definition for me.
When you do that, it will show you the event name "oplGenericV2Trap" which you can use to search the MIB file to find out what the variables that trap uses and then search the variable names to get the definition of them and the values.
If you upload the MIB file to the server, it will create the trap entry in the /etc/snmp/snmptt.conf.nxti file line the example below.
Code: Select all
EVENT oplGenericV2Trap .1.3.6.1.4.1.4184.2.0.2 "Status Events" Normal
FORMAT Received trap "$N" with variables "$+*"
EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N" --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --severity="$s" --uptime="$T" --datetime="$x $X" --unixtime="$@" --category="$c" --bindings="$+*"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "" "The SMS OPENLink Alert process has issued an alert condition. The variables are: $*"
SDESC
The SMS OPENLink Alert process has issued an alert condition. The variables are:oplServiceName - The display name of the NT service on which the alert was issued.oplServiceState - Current state of the NT service.oplDesignName - The ODBC name of the design reporting the alert.oplInterfaceName - The name of the interface associated with the alert (optional).oplConnectionName - The name of the connection associated with the alert (optional).oplAlertName - The alert identifier code.oplAlertSeverity - The severity of the alert.oplAlertDescription - A textual description that elaborates on the condition being reported.oplAlertDescription2 - A continuation of the textual description that elaborates on the condition being reported.oplAlertTime - A textual description that indicates the time when the condition occured.oplAlertExtendedSeverity - Severity, as inidated by the Platform (optional).oplAlertMachine - Machine that initiated this event (optional).oplAlertEntityType - Entity Type the machine belongs to (optional).oplAlertPrimaryStatus - The primary status of the alert.oplAlertSecondaryStatus - The secondary status of the alert.oplAlertURL - The Universal Resource Locator (URL) associated with this alert (optional).oplAlertURL2 - A continuation of the URL associated with this alert (optional).The frequency of this event is determined by two criteria, if the alert is issued as a resultof a situation within the SMS OPENLink server, the individual characteristics of each alertas defined in the OPENLink design database control the frequency. If the event was triggered by an alert request issued through the SMS OPENLink API the frequency is unknownVariables: 1: oplServiceName 2: oplServiceState 3: oplDesignName 4: oplInterfaceName 5: oplConnectionName 6: oplAlertName 7: oplAlertSeverity 8: oplAlertDescription 9: oplAlertDescription2 10: oplAlertTime 11: oplAlertExtendedSeverity 12: oplAlertMachineName 13: oplAlertEntityType 14: oplAlertPrimaryStatus 15: oplAlertSecondaryStatus 16: oplAlertURL 17: oplAlertURL2
EDESC
You can also view the variables there.
Re: SNMP Trap Monitoring in Nagios XI
Posted: Thu Nov 05, 2020 2:27 pm
by emartine
I defined another ok trap see attacehd screenshot. I then saw an item come in as an unconfigured object. I was then told that an issue ocurred with the production system and that the alert that was working is no longer working. snmp trap logs did show the alerts that came through but notification didn't go out from Nagios. SO no one was notified.
Thu Nov 5 08:12:57 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" serverprod - Received trap "oplGenericV2Trap" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4enterprises.4184.2.3.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72.8.83.70.51.51.95.77.77.68:SF33_MMD
5enterprises.4184.2.5.1.0:IN06
6enterprises.4184.2.5.2.0:3
7enterprises.4184.2.5.3.0:EXCESSIVE BACKLOG of transactions.
8enterprises.4184.2.5.4.0:2020-11-05 08:12:57
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"
Thu Nov 5 08:22:57 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" serverprod- Received trap "oplGenericV2Trap"" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4enterprises.4184.2.3.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72.8.83.70.51.51.95.77.77.68:SF33_MMD
5enterprises.4184.2.5.1.0:IN06
6enterprises.4184.2.5.2.0:3
7enterprises.4184.2.5.3.0:EXCESSIVE BACKLOG of transactions.
8enterprises.4184.2.5.4.0:2020-11-05 08:22:57
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"
Thu Nov 5 08:26:27 2020 .1.3.6.1.4.1.4184.2.0.2 Normal "Status Events" serverprod - Received trap "oplGenericV2Trap_Ok" with variables 1"enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4enterprises.4184.2.3.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72.8.83.70.51.51.95.77.77.68:SF33_MMD
5enterprises.4184.2.5.1.0:IN98
6enterprises.4184.2.5.2.0:1
7enterprises.4184.2.5.3.0:Interface alerts paused.
8enterprises.4184.2.5.4.0:2020-11-05 08:26:27
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"
Re: SNMP Trap Monitoring in Nagios XI
Posted: Thu Nov 05, 2020 2:32 pm
by emartine
The other two defined traps are attached
Re: SNMP Trap Monitoring in Nagios XI
Posted: Thu Nov 05, 2020 4:52 pm
by tgriep
If you run a State History report for that Host and service, do you see that the SNMP Traps service received an OK trap as well as a Critical trap?
Re: SNMP Trap Monitoring in Nagios XI
Posted: Thu Nov 05, 2020 5:49 pm
by emartine
No. The critical trap came through and just didn't notify. I went ahead and processed the unconfigured object based on the new definition I created. I then asked him to generate some alerts and I found these which did generate notifications:
The one in bold with 9 lines was an OK state which triggered as critical. I am open to sugestions and how to properly define this.
Thu Nov 5 15:07:34 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" productionserver- Received trap "oplGenericV2Trap" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4enterprises.4184.2.3.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72.7.51.77.95.83.70.51.51:3M_SF33
5enterprises.4184.2.5.1.0:IN13
6enterprises.4184.2.5.2.0:4
7enterprises.4184.2.5.3.0:DOWN, Interface is not operational- ERROR status for Connection.
8enterprises.4184.2.5.4.0:2020-11-05 15:07:34
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"
Thu Nov 5 15:11:10 2020 .1.3.6.1.4.1.4184.2.0.2 Critical "Fatal" productionserver - Received trap "oplGenericV2Trap" with variables "
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4enterprises.4184.2.5.1.0:EN92
5enterprises.4184.2.5.2.0:1
6enterprises.4184.2.5.3.0:RELOAD - Alert process reload by user request.
7enterprises.4184.2.5.4.0:2020-11-05 15:11:10
8enterprises.4184.2.5.8.0:0
9enterprises.4184.2.5.9.0:0"
Other OK states are showing up properly
Thu Nov 5 15:13:11 2020 .1.3.6.1.4.1.4184.2.0.2 Normal "Status Events" productionserver - Received trap "oplGenericV2Trap_Ok" with variables"
1enterprises.4184.2.1.2.1.2.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:Cerner OPENLink 24.1-05
2enterprises.4184.2.1.2.1.5.23.67.101.114.110.101.114.95.79.80.69.78.76.105.110.107.95.50.52.46.49.45.48.53:2
3enterprises.4184.2.2.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72:ICOP25010001H0AH
4nterprises.4184.2.3.2.1.1.16.73.67.79.80.50.53.48.49.48.48.48.49.72.48.65.72.7.76.65.66.95.77.77.68:LAB_MMD
5enterprises.4184.2.5.1.0:IN07
6enterprises.4184.2.5.2.0:1
7enterprises.4184.2.5.3.0:ACTIVE and the Alert process is monitoring this interface.
8enterprises.4184.2.5.4.0:2020-11-05 15:13:10
9enterprises.4184.2.5.8.0:0
10enterprises.4184.2.5.9.0:0"