Page 1 of 1

NSClient, NRPE, check_eventlog

Posted: Tue Oct 06, 2015 10:10 am
by mhixson2
Hi guys,

I'm trying to get some basic event log monitoring set up and I need help with check_eventlog. I've gone through the documentation here but I'm still having trouble. Here's my issue:

Ultimately, I need to watch a custom event log that our application uses for the string "Error reading message on queue". We'll call this log "customlog". An unfiltered check against it says no entries are found in that log.

Code: Select all

$ ./check_nrpe -H [hostname]-p 5666 -t 10 -c check_eventlog -a file=customlog
OK: No entries found|'count'=0;0;5
Filtering it for information messages returns nothing as well (confirmed there are plenty of information messages in the log)

Code: Select all

$ ./check_nrpe -H [hostname] -p 5666 -t 10 -c check_eventlog -a file=customlog "filter=level = 'Information'"
OK: No entries found|'count'=0;0;5
Moving to a different log, system, I run the same commands:

Code: Select all

$ ./check_nrpe -H [hostname] -p 5666 -t 10 -c check_eventlog -a file=system
WARNING: 1/1 System Microsoft-Windows-WindowsUpdateClient (Unable to Connect: Windows is unable to connect to the automatic updates service and therefore cannot download and install updates according to the set schedule. Windows will continue to try to establish a connection.)|'count'=1;0;5
Better, but the filter still fails.

Code: Select all

$ ./check_nrpe -H [hostname] -p 5666 -t 10 -c check_eventlog -a file=system "filter=level = 'Information'"
OK: No entries found|'count'=0;0;5
Trying to filter by age also fails.

Code: Select all

$ ./check_nrpe -H [hostname] -p 5666 -t 10 -c check_eventlog -a file=system "filter=written lt -1d"
OK: No entries found|'count'=0;0;5
Hopefully it's something simple I'm missing.

Oh, and I added this line in nsclient.ini to get the check working. It's the only change I made.

Code: Select all

[/modules]
CheckEventLog=1
Thanks!

Re: NSClient, NRPE, check_eventlog

Posted: Tue Oct 06, 2015 11:26 am
by mhixson2
If the issue here isn't obvious, don't worry about it. I've started writing a powershell script to do this while waiting on an answer and I'm almost done.
If you have an idea, shoot it over and I'll try it. Otherwise, lock it up!
Thanks!

Re: NSClient, NRPE, check_eventlog

Posted: Tue Oct 06, 2015 2:52 pm
by lmiltchev
We will need to do some digging into this. Can you post the nsclient.ini file? Hide sensitive info.
I've started writing a powershell script to do this while waiting on an answer and I'm almost done.
Let us know if the PS script worked out for you. In case it does work as expected, would you mind sharing your script with other users on the forum? Thank you!

Re: NSClient, NRPE, check_eventlog

Posted: Tue Oct 06, 2015 3:01 pm
by mhixson2
Sure, here's the ini file. I'll definitely post the script when it's done.

Code: Select all

[/settings/default]
;A comma separated list of allowed hosts. You can use netmasks (/ syntax) or * to create ranges.
	allowed hosts=[IPs]

[/modules]
;Various system related checks, such as CPU load, process state, service state memory usage and PDH counters.
	CheckSystem=1
;Various file and disk related things.
	CheckDisk=1
;Exactly how it sounds - event log monitoring.
	CheckEventLog=1
;Listens for incoming NRPE connection and processes incoming requests.
	NRPEServer=1
;Execute external scripts
	CheckExternalScripts=1

[/settings/NRPE/server]
	allowed ciphers=ADH
;Allow characters in command definitions
	allow nasty characters=1
;Allow -a arguments in command definitions
	allow arguments=1

[/settings/external scripts]
;Allow characters in external scripts
    allow nasty characters=1
;Allow arguments to be passed to external scripts
	allow arguments=1
	
[/settings/external scripts/scripts]
;Reboot server
	reboot_server=scripts\reboot_server.bat $ARG1$ $ARG2$
;Restart service
	multi_service_restart=scripts\multi_service_restart.bat "$ARG1$"
;Citrix licenses
	citrix_licenses = cmd /c echo scripts\citrix_licenses.ps1; exit($lastexitcode) | powershell.exe -command -
;Cluster resources
	check_mscs=cscript.exe //T:30 //NoLogo scripts\check_microsoft_cluster.vbs $ARG1$ $ARG2$
;Windows update
	windows_update=scripts\windows_update_watcher.bat $ARG1$ $ARG2$

[/settings/log]
;Enable logging
	file name = nsclient-debug-log.txt
	level = debug

Re: NSClient, NRPE, check_eventlog

Posted: Wed Oct 07, 2015 9:19 am
by jdalrymple
I keep handy sample commands around just for this type of a quandary:

Code: Select all

./check_nrpe -H <winhost> -c Check_EventLog -a 'debug' 'warning=count > 0' 'critical=count > 5' "filter=level = 3 AND message like 'none of the' AND log = 'System' AND id in (1014,1015)" 'scan-range=-10m'
WARNING: 0/1 System Microsoft-Windows-DNS-Client (Name resolution for the name mail.google.com timed out after none of the configured DNS servers responded.)|'count'=1;0;5
Beyond that - my suggestion would be to use realtime eventlog monitoring. It's not terribly difficult to set up and is a more sane way of monitoring a log stream.

Re: NSClient, NRPE, check_eventlog

Posted: Wed Oct 07, 2015 1:42 pm
by mhixson2
Cool, thanks! I'm entrenched in using PowerShell as my solution, but I'm going to keep that example and toy with it later.

Here is the PowerShell script I am using. It has been deployed to a few test servers so it's still in a "beta" stage. The only caveat is that my situation requires different events trigger alerts at different levels. I know that one service cannot return both a warning and critical state simultaneously, and I don't want to create two different services for this check since the events that trigger warnings are (as you'd assume) not a huge deal. What I've done is set the logic at the end to check for Critical events and if they exist, only alarm on those. If they don't, continue on to Warning and if they exist, only report on those. If neither exist, an "all is well" message is displayed with an OK status.

If only one event in one log needs monitored, a lot of this script can be stripped away as there is a lot of logic in there dealing with gathering status and perfdata values across all of the events I want to watch.

I'm no PowerShell expert, so I'm always open to ideas/critiques/insults. Here's the script (pardon the substitutions for sensitive stuff):

Code: Select all

#151007

$returnStateOK = 0 
$returnStateWarning = 1
$returnStateCritical = 2
$returnStateUnknown = 3
$critMsg = @()
$warnMsg = @()
$perf = @()
$checkHz = $args[0]
$window = (Get-Date).AddMinutes(-$checkHz)

<# 
The $checkHz (check frequency) variable is passed from the corresponding Nagios service $ARG2$ value
It correlates with the timeframe in the past in which events are read
The logic: This value should match the frequency at which the service is run so that only events created in between runs are read
#>

# Monitor [custom event log name] event log for "Error reading message on queue"

$event1 = Get-EventLog -LogName [custom event log name] | Where-Object {$_.TimeGenerated -gt $window -and $_.Message -like '*Error reading message on queue*'}
$count1 = ($event1).Count
$perf += "queue_errors=$count1;;1"

if ($count1 -ge 1){
    $primary = (($event1).Message).Substring(95,9) | Select -Unique
    $critMsg += "[custom event log name]: Error reading message on queue (primary app: $primary)"
    }

# Monitor Application event log for "Insufficient system resources exist to complete the requested service"

$event2 = Get-EventLog -LogName Application | Where-Object {$_.TimeGenerated -gt $window -and $_.Message -like '*Insufficient system resources exist to complete the requested service*'}
$count2 = ($event2).Count
$perf += "resource_errors=$count2;;1"

if ($count2 -ge 1){
    $warnMsg += "Application Event Log: Insufficient system resources"
    }
        
# Monitor System event log for "operation initiated by the Registry failed"

$event3 = Get-EventLog -LogName System | Where-Object {$_.TimeGenerated -gt $window -and $_.Message -like '*operation initiated by the Registry failed*'}
$count3 = ($event3).Count
$perf += "io_errors=$count3;;1"

if ($count3 -ge 1){
    $critMsg += "System Event Log: I/O Error"
    }

# Monitor [custom event log name] event log for "invalid user"

$event4 = Get-EventLog -LogName [custom event log name] | Where-Object {$_.TimeGenerated -gt $window -and $_.Message -like '*invalid user*'}
$count4 = ($event4).Count
$perf += "user_errors=$count4;;1"

if ($count4 -ge 1){
    $warnMsg += "[custom event log name]: Invalid user found"
    }

# Evaluate, format, and return data to Nagios

if ($critMsg){
    Write-Host CRITICAL: (($critMsg) -join ", ")"|"(($perf) -join " ")
    exit $returnStateCritical
    }
    if ($warnMsg){
        Write-Host WARNING: (($warnMsg) -join ", ")"|"(($perf) -join " ")
        exit $returnStateWarning
        }
        else{
            Write-Host "OK: Event logs are clean|"(($perf) -join " ")
            exit $returnStateOK
            }

Write-Host "UNKNOWN script state"
exit $returnStateUnknown
The output then looks like this (OK and Critical examples)

Code: Select all

CRITICAL: [custom event log name]: Error reading message on queue on primary app server [hostname]| queue_errors=1864;;1 resource_errors=0;;1 io_errors=0;;1 user_errors=0;;1
OK: Event logs are clean| queue_errors=0;;1 resource_errors=0;;1 io_errors=0;;1 user_errors=0;;1
I call the PowerShell external script from a batch where I'm bypassing execution policy, so my line in nsclient.ini looks like this.

Code: Select all

app_eventlogs=scripts\app_eventlogs.bat $ARG1$
And finally, the service looks like this.

Code: Select all

define service {
        host_name                       [hostname]
        service_description             Event logs
        use                             service-template-preprod
        check_command                   check_nrpe_2arg!app_eventlogs!5!!!!!!
        register                        1
        }
Or, from shell on Nagios server

Code: Select all

./check_nrpe -H nchapp049 -t 10 -c app_eventlogs -a 5

Re: NSClient, NRPE, check_eventlog

Posted: Wed Oct 07, 2015 4:47 pm
by tgriep
Thanks for posting your script. I am sure someone will find it helpful.
Is it alright to close the post then?

Re: NSClient, NRPE, check_eventlog

Posted: Thu Oct 08, 2015 7:55 am
by mhixson2
Yes, I am all set for now.

Thanks!