Setting a timeout for a custom script

Engage with the community of users including those using the open source solutions.
Includes Nagios Core, Plugins, and NCPA

Setting a timeout for a custom script

Postby mlazzarotto » Thu Aug 05, 2021 4:37 am

Hello,
for the company I work for, I created a bash script that works as a Nagios plugins.
Without entering too much into details, this script using curl, calls an API and downloads a json with an undefined number of "events" since last 12 hours.
The script, due to the API limitation, can download up to 10.000 events at a time.
If the events since last 12 hours are less than 10.000 the scripts runs one time only. If the events are more than 10.000, the scripts runs recursively and downloads 10k events until the events are all downloaded.

Now, on Nagios the global timeout for plugins is 61 seconds. I calculated that if I have 180k events (in worst case scenario) to downloads, it will require the script to run 18 times. 61/18=3.38. 3.38 seconds is the max time that the single iteration of the script can take to get 10k events.

The problem is: due to some network slowness, I'm afraid that the script couldn't finish to download all the events within the 61 seconds timeout.
Hence the question: can I tell Nagios to increase the timeout only for my plugin and leave the 61 secs for all the others?

Side note: when the timeout kicks-in, Nagios kils the script and leaves the json file = bad. When the script runs again after x minutes, it finds the json file already present and thinks that the previous script is still running.
mlazzarotto
 
Posts: 2
Joined: Wed Apr 28, 2021 4:00 am

Re: Setting a timeout for a custom script

Postby mcapra » Thu Aug 05, 2021 9:38 am

This is not possible. If it's a check being executed via a Nagios command definition, it must honor the service_check_timeout or host_check_timeout.

You may consider having a separate process, orchestrated by something like cron, which is responsible for caching this data to disk somewhere periodically. Then, restructure your Nagios plugin to consume that cached information instead. All you'd really need to add is logic to check the file age to ensure the cache is not too stale.
Former Nagios employee
https://www.mcapra.com/
User avatar
mcapra
 
Posts: 3729
Joined: Thu May 05, 2016 3:54 pm

Re: Setting a timeout for a custom script

Postby mlazzarotto » Thu Aug 05, 2021 9:43 am

I can't recall where, but I read that I can set the check as "passive". In this case the plugin wouldn't be killed by Nagios, correct?
mlazzarotto
 
Posts: 2
Joined: Wed Apr 28, 2021 4:00 am

Re: Setting a timeout for a custom script

Postby mcapra » Thu Aug 05, 2021 10:28 am

Correct, you could also submit the check results passively.
https://assets.nagios.com/downloads/nag ... hecks.html

If NCPA is your weapon of choice:
https://www.nagios.org/ncpa/help/2.2/passive.html

Or if you're still hip to the cron idea, you could pipe those check results directly into the external commands file using either PROCESS_HOST_CHECK_RESULT or PROCESS_SERVICE_CHECK_RESULT:
https://assets.nagios.com/downloads/nag ... mands.html
Former Nagios employee
https://www.mcapra.com/
User avatar
mcapra
 
Posts: 3729
Joined: Thu May 05, 2016 3:54 pm


Return to Community Support

Who is online

Users browsing this forum: Google [Bot] and 17 guests