custom plugin - (Service check timed out after 60.07 seconds

[veeravamsi]

Hi Team,

We have custom plugin which normally takes 3 minutes to complete , so wondering how or whether to increase timeout so that it affects only to this specific plugin.

Which precedence has high priority : ncpa.cfg ( plugin_timeout ) vs --timeout on check_ncpa ... etc.

--Vamsi

[pbroste]

Hello Vamsi,

Thanks for reaching out.


You can add the option -T 120 to the check_ncpa to increase the timeout.

Code: Select all

# 'check_ncpa' command definition
define command {
    command_name    check_ncpa
    command_line    $USER1$/check_ncpa.py -H $HOSTADDRESS$ -T 120 $ARG1$
}

Thanks,
Perry

[benjaminsmith]

Hi Vamsi,

There is a global timeout setting that you may need to increase in the Nagios Core configuration file for service checks.

Format: service_check_timeout=<seconds>
Example: service_check_timeout=60
This is the maximum number of seconds that Nagios will allow service checks to run. If checks exceed this limit, they are killed and a CRITICAL state is returned. A timeout error will also be logged.

There is often widespread confusion as to what this option really does. It is meant to be used as a last ditch mechanism to kill off plugins which are misbehaving and not exiting in a timely manner. It should be set to something high (like 60 seconds or more), so that each service check normally finishes executing within this time limit. If a service check runs longer than this limit, Nagios will kill it off thinking it is a runaway processes.

https://assets.nagios.com/downloads/nag ... gmain.html

Another option would be to set this service up as a passive check and have it send the results to Nagiso XI using NCPA. That would avoid any performance issues with having the global timeout set too high.

https://assets.nagios.com/downloads/ncp ... Checks.pdf

[veeravamsi]

Any other option without having to make changes using NRDP ? as we need to go through Change Process to change the configuration on files and also executing new authentication/ports.

-T 300 still doesnt work for us. Probably global timeout setting is blocking.

[benjaminsmith]

Hi @veeravamsi,

What type of plugin are you running and do you know why it's taking so long to execute? You can check the execution time from the command line. For example,

Code: Select all

time /usr/local/nagios/libexec/check_command <arguments>

One option would be to run the plugin locally on a schedule independently of Nagios, write the results to a log file, and check the log file periodically from Nagios XI. It's not a straight path but that would likely get the job done.

--Benjamin

[veeravamsi]

This is expected to take 150 seconds to query list of things and parse. Wondering why -T and plugin_timeout settings on NCPA config on target servers are not working as expected ?

[benjaminsmith]

Hi,

NCPA has plugin timeout directive as well. You can adjust that in the main ncpa.cfg file on the remote server. However, you would still be subjet to the global timeout in Nagios Core when running check_ncpa.py.

plugin_timeout 60 The plugin execution timeout on the NCPA side. For both active and passive checks. There is also a timeout specified in check_ncpa.py.

See: https://www.nagios.org/ncpa/help/2.0/configuration.html

I would recommend setting this service up as a passive check using NCPA. The guide below has detailed step-by-step instructions.

Using NCPA For Passive Checks

Let me know if that will work for you and if you need any assistance getting this configured.

Regards,
Benjamin

[veeravamsi]

We have increased service_check_timeout as well as plugin_timeout setting ( on target node ) to 300seconds . And why is it still timing out ?

Basic functionality of check_ncpa.py script -T option is to control this timing correct ? if YES , this is not working as expected.

[benjaminsmith]

Hi,

Generally it's not a good idea to increase the default too high so the recommended setup would be passive. If you want to increase the timeouts, this will have to happen in 3 separate places.

NCPA

Code: Select all

plugin_timeout	60	The plugin execution timeout on the NCPA side. For both active and passive checks. There is also a timeout specified in check_ncpa.py.

Both the application timeout and plugin timeout will need to be increased. Lastly, increase the service check timeout in the main Nagios Core configuration file (/usr/local/nagios/etc/nagios.cfg).

Code: Select all

Format:	service_check_timeout=<seconds>
Example:	service_check_timeout=60
This is the maximum number of seconds that Nagios will allow service checks to run. If checks exceed this limit, they are killed and a CRITICAL state is returned. A timeout error will also be logged

Remember to restart both the Nagios Core and NCPA services after making changes to the configuration file.

--Benjamin

[veeravamsi]

I have tried these settings . No luck.

Should i open a support ticket for webex call ?