Nagios Processing Passive Checks in wrong order
Posted: Mon May 20, 2013 7:37 am
Hi all,
I think there is a little issue with the way nagios processes checks sent using nagcmd.
Let me explain. We use SNMPPT to check for LinkDown and LinkUp traps from our switches. If we have a circuit go down and then come up very soon afterwards (like one or two seconds later), SNMPTT seems like it processes the traps correctly and in the right order, but it only spins the directory every 30 seconds. So when traps are processed it will process both a linkdown and linkup trap in rapid succession and correctly fire off the passive check script in the right order. I know this because I have put logging on the script and I can correctly see linkdown and linkup alerts in the right order and they correlate to the same order as I see in snmptrapd log. The problem is that the nagcmd does not seem to be taking the order in which it receives it from the script. Nagios seems to process them randomly, more often than not it process them in the wrong order. Sounds like something within nagios has multiple processes watching nagcmd and if one process is a little slower than the other it then gets processed slower and therefore reported in nagios in the incorrect order.
Has anyone else seen this issue and might know how it can be fixed ?
Thanks
I think there is a little issue with the way nagios processes checks sent using nagcmd.
Let me explain. We use SNMPPT to check for LinkDown and LinkUp traps from our switches. If we have a circuit go down and then come up very soon afterwards (like one or two seconds later), SNMPTT seems like it processes the traps correctly and in the right order, but it only spins the directory every 30 seconds. So when traps are processed it will process both a linkdown and linkup trap in rapid succession and correctly fire off the passive check script in the right order. I know this because I have put logging on the script and I can correctly see linkdown and linkup alerts in the right order and they correlate to the same order as I see in snmptrapd log. The problem is that the nagcmd does not seem to be taking the order in which it receives it from the script. Nagios seems to process them randomly, more often than not it process them in the wrong order. Sounds like something within nagios has multiple processes watching nagcmd and if one process is a little slower than the other it then gets processed slower and therefore reported in nagios in the incorrect order.
Has anyone else seen this issue and might know how it can be fixed ?
Thanks