Processing Liebert UPS traps

[lorejo]

Hi, we have a requirement to receive and alert when our UPS goes on generator power. We have configured the UPS to send traps and we successfully receive them and send out email alerts when them come into XI. Our problem is, every time we transition from street to generator power we receive 9 consecutive traps. All from the same OID within a sub 1 minute time frame. Each trap though has unique text in the 'variable' that we would like to parse so we can either filter and/or pass the string within the variable to our alert to more clearly communicate status.

OID of received trap - .1.3.6.1.4.1.476.1.42.3.7.8.0.1
1. Active:Message:System Input Power Problem
2. Active:Message:Bypass Not Available
3. Active:Message:Bypass Input Voltage Fault
4. Active:Message:Battery Discharging
5. Cleared:Message:System Input Power Problem
6. Cleared:Message:Bypass Not Available
7. Cleared:Message:Bypass Input Voltage Fault
8. Cleared:Message:Battery Discharging
9. Message: System Return to Normal

Example traps from snmptt.log:
1. Tue Oct 16 10:41:14 2018 .1.3.6.1.4.1.476.1.42.3.7.8.0.1 Warning "UPS Events" 10.112.21.111 - Received trap "UPS_Message:Battery_Discharging" with variables "sysUpTime:6:23:37:15.49 enterprises.476.1.42.3.7.7:Message: System Return to Normal"
2. Tue Oct 16 10:41:14 2018 .1.3.6.1.4.1.476.1.42.3.7.8.0.1 Warning "UPS Events" 10.112.21.111 - Received trap "UPS_Message:Battery_Discharging" with variables "sysUpTime:6:23:37:15.46 enterprises.476.1.42.3.7.7:Cleared:Message:Battery Discharging"

An alerting logic example might be:
Alert only when we go on battery and when we come off (Active and Cleared)

Perhaps a plug-in, or something to that effect, already exists.

Thanks in advance.

[ssax]

You would use the MATCH statement on your /etc/snmp/snmptt.conf definition and create multiple definitions for the different states:

http://snmptt.sourceforge.net/docs/snmp ... CONF-MATCH

So say you had this link down trap defined:

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Critical
FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the SNMP entity, acting in $*"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state).  This other state is indicated by the included value
of ifOperStatus.
Variables:
  1: ifIndex
  2: ifAdminStatus
  3: ifOperStatus
EDESC

You would have it match your Return to Normal and set it as OK like this:

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Normal
FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
MATCH $*: (System Return to Normal)
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the SNMP entity, acting in $*"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state).  This other state is indicated by the included value
of ifOperStatus.
Variables:
  1: ifIndex
  2: ifAdminStatus
  3: ifOperStatus
EDESC

Then create another for your Critical notifications:
- Note that if you have multiple MATCH statements it's considered an OR, you can add MATCH MODE=and to set it to an AND style of matching

Code: Select all

EVENT linkDown .1.3.6.1.6.3.1.1.5.3 "Status Events" Critical
FORMAT A linkDown trap signifies that the SNMP entity, acting in $*
MATCH $*: (Battery Discharging)
MATCH $*: (Bypass Input Voltage Fault)
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "A linkDown trap signifies that the SNMP entity, acting in $*"
SDESC
A linkDown trap signifies that the SNMP entity, acting in
an agent role, has detected that the ifOperStatus object for
one of its communication links is about to enter the down
state from some other state (but not from the notPresent
state).  This other state is indicated by the included value
of ifOperStatus.
Variables:
  1: ifIndex
  2: ifAdminStatus
  3: ifOperStatus
EDESC

Let us know if you have any questions, if you need help, please attach your /etc/snmp/snmptt.conf file.

[lorejo]

Thank you very much. Under SNMP Trap Interface we see the traps with the warning first and the normal shortly after. Unfortunately we only get email notifications for the warning events. Additionally, the SNMP Traps service for this host stays in a 'Warning State'. Is there additional notification configuration settings that I need to enter in CCM?

Here is the entry in snmptt,conf.nxti -
EVENT UPS-Restored .1.3.6.1.4.1.476.1.42.3.7.8.0.1 "UPS Events" Normal
FORMAT Received trap "$N" with variables "$+*"
EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N" --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --bindings="$+*"
EXEC /usr/local/bin/snmptraphandling.py "$aR" "SNMP Traps" OK $@ "" "This is a trap from the UPS that is used to toggle the trap receiver to normal state"
MATCH $*: (System Return to Normal)
SDESC
This trap is used to toggle the receiver to a normal state.
EDESC


EVENT UPS_Message:Battery_Discharging .1.3.6.1.4.1.476.1.42.3.7.8.0.1 "UPS Events" Warning
FORMAT Received trap "$N" with variables "$+*"
EXEC php /usr/local/nagiosxi/scripts/nxti.php --event_name="$N" --event_oid="$i" --numeric_oid="$o" --symbolic_oid="$O" --community="$C" --trap_hostname="$R" --trap_ip="$aR" --agent_hostname="$A" --agent_ip="$aA" --category="$c" --severity="$s" --uptime="$T" --datetime="$x $X" --bindings="$+*"
EXEC /usr/local/bin/snmptraphandling.py "$aR" "SNMP Traps" WARNING $@ "" "SNMP Trap Received alerting that Houston UPS system transitioned to Battery Power"
EXEC php /usr/local/nagios/libexec/nagios-ups-mas.php 2
MATCH $*: (Active:Message:Battery Discharging)
SDESC
UPS_Message:Battery_Discharging
EDESC

Thanks in advance.

[ssax]

Please post your service definition from /usr/local/nagios/var/objects.cache so that we can see how you have it set up.

Then run this command and PM me the resulting /tmp/SNMPFILES.zip file:

Code: Select all

zip -r /tmp/SNMPFILES.zip /etc/snmp /usr/share/snmp/mibs

[lorejo]

Thanks for your quick response. Here is the requested service definition:

define service {
host_name HTX-UPS-01
service_description SNMP Traps
check_period xi_timeperiod_24x7
check_command check_dummy!0!"TRAP RESET"
contacts thazlewood,jolore
contact_groups admins
notification_period xi_timeperiod_24x7
initial_state o
importance 0
check_interval 1.000000
retry_interval 1.000000
max_check_attempts 3
is_volatile 1
parallelize_check 1
active_checks_enabled 0
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 0
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options a
notifications_enabled 1
notification_interval 1.000000
first_notification_delay 0.000000
stalking_options o,w,u,c
process_perf_data 1
icon_image snmptrap.png
retain_status_information 1
retain_nonstatus_information 1
_XIWIZARD snmp_trap
}

I have attached the zip file as requested as well.

Regards,
Josh...

[ssax]

If you want the trap to reset to an OK status on it's own you can go to Configure > Core Config Manager > Services:
- Edit the SNMP Traps service
- Click the Check Settings tab
- Set Active Checks Enabled to On
- Save
- Apply Config

Now it will be reset to OK automatically every X minutes, where X is the check_interval on the service (currently 1 minute).

You should have received the recovery based on your information posted (unless the users disabled recoveries in their notification preferences) which leads me to believe you may be running a version of the backend Core system that has a bug with recoveries.

What version of XI are you using? You can grab it from the bottom left hand side of the web interface.

Also, please send the output of this command:

Code: Select all

/usr/local/nagios/bin/nagios -V