Page 1 of 1

Nagios XI 5.5.2 and poweshell script timeout

Posted: Tue Sep 18, 2018 6:57 am
by Wojtek
Hi!

I have problem with timeout with powershell script. On 90% servers I don't have problem but on some of them I have timeout (Service check timed out after 60.01 seconds).

It's my nsclient.ini

Code: Select all

# If you want to fill this file with all avalible options run the following command:
#   nscp settings --generate --add-defaults --load-all
# If you want to activate a module and bring in all its options use:
#   nscp settings --activate-module <MODULE NAME> --add-defaults
# For details run: nscp settings --help


; Undocumented section
[/settings/default]

; Undocumented key
password = xxxx

; Undocumented key
allowed hosts = 127.0.0.1,::1,x.x.x.x

; CACHE ALLOWED HOSTS - If host names (DNS entries) should be cached, improves speed and security somewhat but won't allow you to have dynamic IPs for your Nagios server.
cache allowed hosts = 1

; TIMEOUT - Timeout when reading packets on incoming sockets. If the data has not arrived within this time we will bail out.
timeout = 120


; Undocumented section
[/settings/NRPE/server]

; Undocumented key
ssl options = no-sslv2,no-sslv3

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
use ssl = 1

; PORT NUMBER - Port to use for NRPE.
port = 5666

; EXTENDED RESPONSE - Send more then 1 return packet to allow response to go beyond payload size (requires modified client if legacy is true this defaults to false).
extended response = 1

; COMMAND ARGUMENT PROCESSING - This option determines whether or not the we will allow clients to specify arguments to commands that are executed.
allow arguments = true

; COMMAND ALLOW NASTY META CHARS - This option determines whether or not the we will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow nasty characters = true

; ALLOW INSECURE CHIPHERS and ENCRYPTION - Only enable this if you are using legacy check_nrpe client.
insecure = true

; VERIFY MODE - Comma separated list of verification flags to set on the SSL socket.  none	The server will not send a client certificate request to the client, so the client will not send a certificate. peer	The server sends a client certificate request to the client and the certificate returned (if any) is checked. fail-if-no-cert	if the client did not return a certificate, the TLS/SSL handshake is immediately terminated. This flag must be used together with peer. peer-cert	Alias for peer and fail-if-no-cert. workarounds	Various bug workarounds. single	Always create a new key when using tmp_dh parameters. client-once	Only request a client certificate on the initial TLS/SSL handshake. This flag must be used together with verify-peer   
verify mode = none


; Undocumented section
[/modules]

; Undocumented key
CheckHelpers = 1

; Undocumented key
CheckEventLog = 1

; Undocumented key
CheckNSCP = 1

; Undocumented key
CheckDisk = 1

; Undocumented key
CheckSystem = 1

; Undocumented key
NSClientServer = 1

; NRPEServer - A server that listens for incoming NRPE connection and processes incoming requests.
NRPEServer = enabled

; CheckExternalScripts - Execute external scripts
CheckExternalScripts = enabled


; Configure which services has to be in which state
[/settings/system/windows/service mapping]


; A set of options to configure the real time checks
[/settings/system/windows/real-time]


; Add counters to check
[/settings/system/windows/counters]


; Section for system checks and system settings
[/settings/system/windows]

; DEFAULT LENGTH - Used to define the default interval for range buffer checks (ie. CPU).
default buffer length = 1h


; Configure log file properties.
[/settings/log/file]

; MAXIMUM FILE SIZE - When file size reaches this it will be truncated to 50% if set to 0 (default) truncation will be disabled
max size = 0


; Section for configuring the log handling.
[/settings/log]

; LOG LEVEL - Log level to use. Available levels are error,warning,info,debug,trace
level = info

; DATEMASK - The size of the buffer to use when getting messages this affects the speed and maximum size of messages you can recieve.
date format = %Y-%m-%d %H:%M:%S

; FILENAME - The file to write log data to. Set this to none to disable log to file.
file name = ${exe-path}/nsclient.log


; A set of options to configure the real time checks
[/settings/eventlog/real-time]

; REAL TIME CHECKING - Spawns a background thread which detects issues and reports them back instantly.
enabled = 0

; STARTUP AGE - The initial age to scan when starting NSClient++
startup age = 30m

; LOGS TO CHECK - Comma separated list of logs to check
log = application,system

; DEBUG - Log missed records (useful to detect issues with filters) not useful in production as it is a bit of a resource hog.
debug = 0


; Section for configuring the shared session.
[/settings/shared session]

; ENABLE THE SAHRED SESSION - This is currently not added in 0.4.x
enabled = false


; Section for configuring the crash handler.
[/settings/crash]

; RESTART SERVICE NAME - The url to submit crash reports to
restart target = NSCP

; CRASH ARCHIVE LOCATION - The folder to archive crash dumps in
archive folder = ${shared-path}/crash-dumps

; SUBMISSION URL - The url to submit crash reports to
submit url = https://crash.nsclient.org/post

; RESTART - Submit crash reports to nsclient.org (or your configured submission server)
restart = true

; ARCHIVE CRASHREPORTS - Archive crash reports in the archive folder
archive = true

; SUBMIT CRASHREPORTS - Submit crash reports to nsclient.org (or your configured submission server)
submit = false


; A set of filters to use in real-time mode
[/settings/system/windows/real-time/checks]


; Section for NSClient (NSClientServer.dll) (check_nt) protocol options.
[/settings/NSClient/server]

; PERFORMANCE DATA - Send performance data back to Nagios (set this to 0 to remove all performance data).
performance data = 1

; ENABLE SSL ENCRYPTION - This option controls if SSL should be enabled.
use ssl = 0

; PORT NUMBER - Port to use for check_nt.
port = 12489


; Section for external scripts configuration options (CheckExternalScripts).
[/settings/external scripts]

; COMMAND TIMEOUT - The maximum time in seconds that a command can execute. (if more then this execution will be aborted). NOTICE this only affects external commands not internal ones.
timeout = 120

; COMMAND ARGUMENT PROCESSING - This option determines whether or not the we will allow clients to specify arguments to commands that are executed.
allow arguments = true

; COMMAND ALLOW NASTY META CHARS - This option determines whether or not the we will allow clients to specify nasty (as in |`&><'"\[]{}) characters in arguments.
allow nasty characters = true


; Files to be included in the configuration
[/includes]


; Section for the EventLog Checker (CheckEventLog.dll).
[/settings/eventlog]

; BUFFER_SIZE - The size of the buffer to use when getting messages this affects the speed and maximum size of messages you can recieve.
buffer size = 131072

; LOOKUP NAMES - Lookup the names of eventlog files
lookup names = 1

; DEBUG - Log more information when filtering (useful to detect issues with filters) not useful in production as it is a bit of a resource hog.
debug = 0


; A list of scripts available to run from the CheckExternalScripts module. Syntax is: <command>=<script> <arguments>
[/settings/external scripts/scripts]
check_updates=cmd /c echo scripts\Check-Updates.ps1; exit $LastExitCode | powershell.exe -command -

; A list of aliases available.
;An alias is an internal command that has been predefined to provide a single command without arguments. Be careful so you don't create loops (ie check_loop=check_a, check_a=check_loop)
[/settings/external scripts/alias]

; alias_sched_all - Alias for alias_sched_all. To configure this item add a section called: /settings/external scripts/alias/alias_sched_all
alias_sched_all = check_tasksched show-all "syntax=${title}: ${exit_code}" "crit=exit_code ne 0"

; alias_file_size - Alias for alias_file_size. To configure this item add a section called: /settings/external scripts/alias/alias_file_size
alias_file_size = check_files "path=$ARG1$" "crit=size > $ARG2$" "top-syntax=${list}" "detail-syntax=${filename] ${size}" max-dir-depth=10

; alias_process_hung - Alias for alias_process_hung. To configure this item add a section called: /settings/external scripts/alias/alias_process_hung
alias_process_hung = check_process "filter=is_hung" "crit=count>0"

; alias_process - Alias for alias_process. To configure this item add a section called: /settings/external scripts/alias/alias_process
alias_process = check_process "process=$ARG1$" "crit=state != 'started'"

; alias_service_ex - Alias for alias_service_ex. To configure this item add a section called: /settings/external scripts/alias/alias_service_ex
alias_service_ex = check_service "exclude=Net Driver HPZ12" "exclude=Pml Driver HPZ12" exclude=stisvc

; alias_event_log - Alias for alias_event_log. To configure this item add a section called: /settings/external scripts/alias/alias_event_log
alias_event_log = check_eventlog

; alias_volumes_loose - Alias for alias_volumes_loose. To configure this item add a section called: /settings/external scripts/alias/alias_volumes_loose
alias_volumes_loose = check_drivesize

; alias_volumes - Alias for alias_volumes. To configure this item add a section called: /settings/external scripts/alias/alias_volumes
alias_volumes = check_drivesize

; alias_disk - Alias for alias_disk. To configure this item add a section called: /settings/external scripts/alias/alias_disk
alias_disk = check_drivesize

; alias_up - Alias for alias_up. To configure this item add a section called: /settings/external scripts/alias/alias_up
alias_up = check_uptime

; alias_file_age - Alias for alias_file_age. To configure this item add a section called: /settings/external scripts/alias/alias_file_age
alias_file_age = check_files "path=$ARG1$" "crit=written > $ARG2$" "top-syntax=${list}" "detail-syntax=${filename] ${written}" max-dir-depth=10

; alias_cpu_ex - Alias for alias_cpu_ex. To configure this item add a section called: /settings/external scripts/alias/alias_cpu_ex
alias_cpu_ex = check_cpu "warn=load > $ARG1$" "crit=load > $ARG2$" time=5m time=1m time=30s

; alias_process_stopped - Alias for alias_process_stopped. To configure this item add a section called: /settings/external scripts/alias/alias_process_stopped
alias_process_stopped = check_process "process=$ARG1$" "crit=state != 'stopped'"

; alias_service - Alias for alias_service. To configure this item add a section called: /settings/external scripts/alias/alias_service
alias_service = check_service

; alias_cpu - Alias for alias_cpu. To configure this item add a section called: /settings/external scripts/alias/alias_cpu
alias_cpu = check_cpu

; alias_mem - Alias for alias_mem. To configure this item add a section called: /settings/external scripts/alias/alias_mem
alias_mem = check_memory

; alias_process_count - Alias for alias_process_count. To configure this item add a section called: /settings/external scripts/alias/alias_process_count
alias_process_count = check_process "process=$ARG1$" "warn=count > $ARG2$" "crit=count > $ARG3$"

; alias_sched_long - Alias for alias_sched_long. To configure this item add a section called: /settings/external scripts/alias/alias_sched_long
alias_sched_long = check_tasksched "filter=status = 'running'" "detail-syntax=${title} (${most_recent_run_time})" "crit=most_recent_run_time < -$ARG1$"

; alias_sched_task - Alias for alias_sched_task. To configure this item add a section called: /settings/external scripts/alias/alias_sched_task
alias_sched_task = check_tasksched show-all "filter=title eq '$ARG1$'" "detail-syntax=${title} (${exit_code})" "crit=exit_code ne 0"

; alias_disk_loose - Alias for alias_disk_loose. To configure this item add a section called: /settings/external scripts/alias/alias_disk_loose
alias_disk_loose = check_drivesize


; A list of wrapped scripts (ie. scruts using a template mechanism). The template used will be defined by the extension of the script.
[/settings/external scripts/wrapped scripts]


; A list of templates for wrapped scripts.
;%SCRIPT% will be replaced by the actual script an %ARGS% will be replaced by any given arguments.
[/settings/external scripts/wrappings]

; BATCH FILE WRAPPING - 
bat = scripts\\%SCRIPT% %ARGS%

; VISUAL BASIC WRAPPING - 
vbs = cscript.exe //T:30 //NoLogo scripts\\lib\\wrapper.vbs %SCRIPT% %ARGS%

; POWERSHELL WRAPPING - 
ps1 = cmd /c echo scripts\\%SCRIPT% %ARGS%; exit($lastexitcode) | powershell.exe -command -


; A set of filters to use in real-time mode
[/settings/eventlog/real-time/filters]


[/paths]

; Path for shared-path - 
shared-path = C:\Program Files\NSClient++

; Path for exe-path - 
exe-path = C:\Program Files\NSClient++

; Path for crash-folder - 
crash-folder = C:\Program Files\NSClient++

; Path for certificate-path - 
certificate-path = ${shared-path}/security

; Path for base-path - 
base-path = C:\Program Files\NSClient++

; Path for module-path - 
module-path = ${shared-path}/modules
I set timeout =120 and my comand is $USER1$/check_nrpe -H $HOSTADDRESS$ -t 240 -c check_updates $ARG1$ $ARG2$

I use nsclient NSCP-0.4.4.23

I set different timeout but on some servers problem is still existing.

Can I add anything else?

BR
Wojtek

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Tue Sep 18, 2018 2:44 pm
by npolovenko
@Wojtek, On the nagios server please open the /usr/local/nagios/etc/nrpe.cfg file and change the "command_timeout" to 120:
# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60
Then restart the nagios process with:
service nagios restart

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Wed Sep 19, 2018 4:27 am
by Wojtek
Hi!

I've changed

# COMMAND TIMEOUT
# This specifies the maximum number of seconds that the NRPE daemon will
# allow plugins to finish executing before killing them off.

command_timeout=60

to 120 and restart all Nagios Server but problem still exist. What else I can change?

BR
Wojtek

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Wed Sep 19, 2018 4:21 pm
by scottwilkerson
Is it realistic that these commands need more than 60 seconds to run? This usually isn't the case and is caused by the servers not getting the packets due to network issues , etc, but in some instances if you do have long running plugins on the remore server you may need to increase the timeout in the nagios.cfg on the Nagios server

Code: Select all

service_check_timeout=120
then restart nagios

Code: Select all

service nagios restart
this would give the checks up to 2 minutes on the nagios side.

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Wed Sep 19, 2018 4:22 pm
by npolovenko
@Wojtek, Are you using the same version of the NSCLient on all servers? Can you upload the nsclient.log file?

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Thu Sep 20, 2018 5:35 am
by Wojtek
Thanks for response.

Script has quit long time running because it is for checking Windows updates on hosts. Sometimes 60 seconds is sufficient but on some servers not.

I changed two things.

I add two lines in /usr/local/nagios/etc/nrpe.cfg

service_check_timeout=120
host_check_timeout=120

but after service nagios restart problem still existed.

When I change lines service_check_timeout=120 and service nagios restart problem was solved.

Thanks for assistance.

BR
Wojtek

Re: Nagios XI 5.5.2 and poweshell script timeout

Posted: Thu Sep 20, 2018 2:21 pm
by npolovenko
@Wojtek, Thanks for the update. I will be closing this thread as resolved.