Check for strings in a log file - disable autoclear?

[steph280]

I currently have Nagios setup to run (through check_nrpe) check_log3.exe on windows 2008 server to report back any instance of strings "CRITICAL" in an application log file.

Things are working fine, except the following scenario:

When a "CRITICAL" shows up in the log file, check_log3 would correctly report the logfile line back to Nagios, and shows up as alert in Nagios dashboard. But as soon as a new line (that doesn't contain "CRITICAL") is appended to the application log file, this alert in Nagios dashboard would auto-clear itself. I am assuming this is because check_log3.exe didn't find a match to "CRITICAL" and thus returns "OK: Found 0 lines (limit=1/0): No Matches found".

Is there a way to prevent it from auto-clearing? I want somebody to check on this and manually acknowledge the alarm before it clears off from Nagios dashboard.

Thanks in advance.

Stephen

[ssax]

The only method that I can think of for you to achieve that functionality would be to use passive checks and not submit OK results from your passive check, only submit the criticals, that way the state won't change from critical automatically.

Please read more about passive checks here:

Code: Select all

https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/passivechecks.html

[steph280]

we decided to just forward the result of check_log3.exe to a netcool server as soon as it detects a CRITICAL text string in the log file. This is done using a custom "send-service-trap" event handler that sends the alarm via SNMP trap. This is working fine, alarms are being sent to Netcool, but with the wrong content.

In the Nagios event log we see the alert and the recovery event:
Service Recovery 2016-08-26 14:50:30 SERVICE ALERT: loggerhost1;log check;OK;SOFT;2;OK: Found 0 lines (limit=1/1): No matches found.
Service Critical 2016-08-26 14:49:31 SERVICE ALERT: loggerhost1;log check;CRITICAL;SOFT;1;CRITICAL: Found 2 lines (limit=1/1): error event detected CRITICAL from environment sensor A100

For some odd reason the event that got sent to Netcool with a CRITICAL alarm is "OK;SOFT;2;OK: Found 0 lines (limit=1/1): No matches found." As if it skipped the Critical event's message and used the recovery message instead.

Any idea what may be causing this? If there a setting somewhere that waits for 2 messages before sending?

thanks again!

[gormank]

A sticky alert feature would be nice...

[ssax]

Please post your send-service-trap script, your command definition, and your service definition so that we can review how it's set up.

Thank you

[steph280]

send-service-trap

# Arguments:
# $1 = Management Station
# $2 = Community String
# $3 = host_name
# $4 = service_description (Description of the service)
# $5 = return_code (An integer that determines the state
# of the service check, 0=OK, 1=WARNING, 2=CRITICAL,
# 3=UNKNOWN).
# $6 = plugin_output (A text string that should be used
# as the plugin output for the service check)
#
#

/usr/bin/snmptrap -D TOKEN[ALL] -v 2c -c $2 $1 '' NAGIOS-NOTIFY-MIB::nSvcEvent nSvcHostname s "$3" nSvcDesc s "$4" nSvcStateID i $5 nSvcOutput s "$6"

command definition:

/usr/local/nagios/libexec/eventhandlers/send-service-trap 135.163.100.100 "$HOSTNAME$" "$SERVICEDESC$" $SERVICESTATEID$ "$SERVICEOUTPUT$"

The service definition:

nagios1.jpg

nagios2.jpg

Thanks again!

Stephen

[steph280]

Also the External Scripts entry in the NSC.conf file from the client being monitored:

check_log3 = scripts\check_log3.exe -l "c:\logs\today.log" -p CRITICAL|MAJOR -c 1

[ssax]

Your event handler will run every state change, including soft and recoveries.

You need to make your script smarter, please see here at the bottom for an example that you can use to make your script understand if it's a HARD/SOFT and what state it's in so that you will only send when you want:

https://assets.nagios.com/downloads/nag ... dlers.html