UDP timeouts happening with SNMP checks

[rzietlow]

Hi Support staff,

Wondering if you can assist/know of a sysctl to tune for UDP timeouts.

Currently we monitor a lot of our systems via snmp, this includes windows systems. in the Nagios dashboard we are seeing “ERROR: General Time-out (Alarm Signal)” Service problem. I did a tcpdump of the traffic. I see the Nagios server query the host, but then about 25-30 seconds later we are seeing a response from the server. When we look at the server the CPU is pegged, so this makes some sense, but we still need to know that the server is consuming a large amount of CPU.

My tcpdump shows 2 queries. The first one comes back 24 seconds later and we get “Port Unreachable” This query is related as the src & dst ports match. The 2nd query we get Host administratively prohibited (iptables blocking it because it’s timeout was initially set to 30 seconds) I have tweaked the iptables timeout, but I cannot seem to find one for the udp traffic

we're on the VA running CentOS release 6.9 (Final) and nagios 5.4.13

[scottwilkerson]

What is the full command your check is running? By default SNMP checks have a timeout of 10 seconds if a higher timeout isn't specified.

[tgriep]

I found these options you can add to the sysctl config file to increase the timeout settings for UDP packets.

nf_conntrack_udp_timeout - INTEGER (seconds)
default 30

nf_conntrack_udp_timeout_stream - INTEGER (seconds)
default 180

This extended timeout will be used in case there is an UDP stream
detected.

Those options were found here.

https://www.kernel.org/doc/Documentatio ... sysctl.txt



Another thing to do is to increase the timeout value for the plugin.
If you edit the SNMP commands and add the following to the command line

Code: Select all

-t 60

That will increase the timeout of the plugin to 60 seconds.
This would have to be done as well.

[rzietlow]

the conntrack ones aren't working, that's for iptables (it's default 30 in my tcpdump if it's less than 30 i see just port unreachable, if over 30 I get admin prohibited..once I updated that setting it was just unreachable)

current check is check_xi_service_snmp_win_storage! -C $COMMUNITY --v2c -m ^C: -w 85 -c 90 -f

i'll add the -t for a host, any simple way to add globally to the already existing checks (if this works?)

[tgriep]

Edit the command called check_xi_service_snmp_win_storage and add it there. Then all services that use that command will have the -t option updated.

[rzietlow]

I added a -t 60 and it went green once and back to general timeout.

when i do a tcpdump it takes about 45 seconds to respond and i'm still seeing ICMP port unreachable. (not getting admin prohibited, which matches the conntrack sysctl we set)

i'll do more digging on exactly, but other help would be appreciated.

[tgriep]

I think I found the UDP timeout options you are looking for.
ip_ct_udp_timeout and ip_ct_udp_timeout_stream

The description of those options at at this link.
https://www.frozentux.net/ipsysctl-tuto ... rence.html

Let us know if increasing these settings fixes the issue for you.

[rzietlow]

ok, so I'll have to patch the kernel. was hoping there would be a simpler fix than that. I will dig a little more and possibly not have the checks rely on snmp.

will keep posted.

[cdienger]

Thanks for the update!