Hi all,
I run a small AI service (an ad generator, ravvi.ai) on a handful of Ubuntu boxes with two GPU workers behind an nginx front end. Core Nagios has been fine for host/disk/load, but the failure mode that actually hurts us is not a dead host - it is the render queue silently backing up while every host check stays green.
Current setup:
- check_http on the public endpoint (200 + string match)
- check_nrpe for load / disk / memory on the workers
- nvidia-smi wrapped in a shell script for GPU temp, called via NRPE
What I have not solved:
1. Queue depth lives in Redis. Is the accepted pattern here a custom plugin that runs redis-cli LLEN and returns the usual 0/1/2 exit codes with perfdata, or do people push this through NSCA/NRDP as a passive check instead? I would rather not have every poll open a Redis connection if there is a cleaner way.
2. I want to alert on "p95 render time over 90s for 10 minutes", not on a single bad sample. Is flap detection plus a longer max_check_attempts the right tool for that, or should the smoothing happen inside the plugin?
3. For the GPU workers, is there a maintained plugin for nvidia-smi that I have missed on Nagios Exchange? The scripts I found are mostly 2019 and older and none of them emit perfdata in a form Nagios graphing likes.
Happy to post my current command definitions if that helps. Any pointers appreciated.