Page 1 of 2
custom plugin - (Service check timed out after 60.07 seconds
Posted: Mon Jul 12, 2021 9:31 am
by veeravamsi
Hi Team,
We have custom plugin which normally takes 3 minutes to complete , so wondering how or whether to increase timeout so that it affects only to this specific plugin.
Which precedence has high priority : ncpa.cfg ( plugin_timeout ) vs --timeout on check_ncpa ... etc.
--Vamsi
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Mon Jul 12, 2021 1:26 pm
by pbroste
Hello Vamsi,
Thanks for reaching out.
You can add the option -T 120 to the check_ncpa to increase the timeout.
Code: Select all
# 'check_ncpa' command definition
define command {
command_name check_ncpa
command_line $USER1$/check_ncpa.py -H $HOSTADDRESS$ -T 120 $ARG1$
}
Thanks,
Perry
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Mon Jul 12, 2021 4:31 pm
by benjaminsmith
Hi Vamsi,
There is a global timeout setting that you may need to increase in the Nagios Core configuration file for service checks.
Format: service_check_timeout=<seconds>
Example: service_check_timeout=60
This is the maximum number of seconds that Nagios will allow service checks to run. If checks exceed this limit, they are killed and a CRITICAL state is returned. A timeout error will also be logged.
There is often widespread confusion as to what this option really does. It is meant to be used as a last ditch mechanism to kill off plugins which are misbehaving and not exiting in a timely manner. It should be set to something high (like 60 seconds or more), so that each service check normally finishes executing within this time limit. If a service check runs longer than this limit, Nagios will kill it off thinking it is a runaway processes.
https://assets.nagios.com/downloads/nag ... gmain.html
Another option would be to set this service up as a passive check and have it send the results to Nagiso XI using NCPA. That would avoid any performance issues with having the global timeout set too high.
https://assets.nagios.com/downloads/ncp ... Checks.pdf
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Thu Jul 15, 2021 9:35 am
by veeravamsi
Any other option without having to make changes using NRDP ? as we need to go through Change Process to change the configuration on files and also executing new authentication/ports.
-T 300 still doesnt work for us. Probably global timeout setting is blocking.
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Thu Jul 15, 2021 3:43 pm
by benjaminsmith
Hi
@veeravamsi,
What type of plugin are you running and do you know why it's taking so long to execute? You can check the execution time from the command line. For example,
Code: Select all
time /usr/local/nagios/libexec/check_command <arguments>
One option would be to run the plugin locally on a schedule independently of Nagios, write the results to a log file, and check the log file periodically from Nagios XI. It's not a straight path but that would likely get the job done.
--Benjamin
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Fri Jul 16, 2021 8:10 am
by veeravamsi
This is expected to take 150 seconds to query list of things and parse. Wondering why -T and plugin_timeout settings on NCPA config on target servers are not working as expected ?
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Fri Jul 16, 2021 12:45 pm
by benjaminsmith
Hi,
NCPA has plugin timeout directive as well. You can adjust that in the main ncpa.cfg file on the remote server. However, you would still be subjet to the global timeout in Nagios Core when running check_ncpa.py.
plugin_timeout 60 The plugin execution timeout on the NCPA side. For both active and passive checks. There is also a timeout specified in check_ncpa.py.
See:
https://www.nagios.org/ncpa/help/2.0/configuration.html
I would recommend setting this service up as a passive check using NCPA. The guide below has detailed step-by-step instructions.
Using NCPA For Passive Checks
Let me know if that will work for you and if you need any assistance getting this configured.
Regards,
Benjamin
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Thu Jul 22, 2021 11:31 am
by veeravamsi
We have increased service_check_timeout as well as plugin_timeout setting ( on target node ) to 300seconds . And why is it still timing out ?
Basic functionality of check_ncpa.py script -T option is to control this timing correct ? if YES , this is not working as expected.
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Thu Jul 22, 2021 3:00 pm
by benjaminsmith
Hi,
Generally it's not a good idea to increase the default too high so the recommended setup would be passive. If you want to increase the timeouts, this will have to happen in 3 separate places.
NCPA
Code: Select all
plugin_timeout 60 The plugin execution timeout on the NCPA side. For both active and passive checks. There is also a timeout specified in check_ncpa.py.
Both the application timeout and plugin timeout will need to be increased. Lastly, increase the service check timeout in the main Nagios Core configuration file (/usr/local/nagios/etc/nagios.cfg).
Code: Select all
Format: service_check_timeout=<seconds>
Example: service_check_timeout=60
This is the maximum number of seconds that Nagios will allow service checks to run. If checks exceed this limit, they are killed and a CRITICAL state is returned. A timeout error will also be logged
Remember to restart both the Nagios Core and NCPA services after making changes to the configuration file.
--Benjamin
Re: custom plugin - (Service check timed out after 60.07 sec
Posted: Fri Jul 23, 2021 3:15 am
by veeravamsi
I have tried these settings . No luck.
Should i open a support ticket for webex call ?