The DNS servers at a sister company went down yesterday and among the other arm flailing madness that ensued was a very curious Nagios plugins (1.4.15) problem. Despite the DNS search orders in our linux config starting with our DNS servers and finishing with that particular sister companies check_nt and check_nrpe (most noticeably) were timing out (taking longer than 10 seconds to reply) for every server.
If we were to run any native linux command on the Nagios box name resolution was instantaneous as expected (ping, ssh, telnet, etc). The minute we tried check_nt with a hostname or even more curiously an FQDN it would take somewhere between 15 ~ 25 seconds to respond. The only plausible reason I can think of for this is that for some reason it's doing the DNS lookup order backwards. What further cements this idea is that the Nagios XI server was completely unaffected but does not have their DNS in its resolv.conf.
It's not really a catastrophic issue... but it might be worth investigating.
Possible nagios plugins bug
Re: Possible nagios plugins bug
Would you mind posting this to the nagios-plugins or the nagios plugins devel mailing lists on source forge? The plugins pack is actually a community maintained project, and this seems like a question best suited for the plugin developers. I think they're going to have a better idea as to what's going on there than we would.