Check_SNMP CRITICAL - Plugin timed out while executing syste

[OptimusB]

All of a sudden today, all checks using check_snmp seems to be failing and returning error "CRITICAL - Plugin timed out while executing system call". On some checks I am getting the error "(Service check timed out after 60.01 seconds)"
A browse through the forums mentioned a couple checks. So I've done the following:

1. Ran snmpwalk on the devices and it is returning proper values without issue.
2. Other snmp related check commands are working: ie. check_snmp_storage_wizard, check_snmp_load_wizard, etc.
3. ran check_snmp manually and it is also returning a timeout error

Code: Select all

./check_snmp -H server-P 2c -C public -o enterprises.789.1.5.4.1.6.1654 -t 30 -vvv
/usr/bin/snmpget -Le -t 30 -r 5 -m ALL -v 2c [authpriv] server:161 enterprises.789.1.5.4.1.6.1654
CRITICAL - Plugin timed out while executing system call

./check_snmp -H server -P 2c -C public -o enterprises.789.1.5.4.1.6.1654 -t 5 -vvv
/usr/bin/snmpget -Le -t 5 -r 5 -m ALL -v 2c [authpriv] server:161 enterprises.789.1.5.4.1.6.1654
CRITICAL - Plugin timed out while executing system call

./check_snmp -H server -P 2c -C public -o enterprises.789.1.5.4.1.6.1654 -vvv
/usr/bin/snmpget -Le -t 1 -r 5 -m ALL -v 2c [authpriv] server:161 enterprises.789.1.5.4.1.6.1654
CRITICAL - Plugin timed out while executing system call

Code: Select all

snmpwalk -c public -v2c server
SNMPv2-MIB::sysDescr.0 = STRING: NetApp Release 8.2.1 Cluster-Mode: Fri Mar 21 18:39:00 PDT 2014
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.789.2.5
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (179630315) 20 days, 18:58:23.15
SNMPv2-MIB::sysContact.0 = STRING:

Any ideas? Thanks.

[Box293]

Are there any hung snmp processes running?

Code: Select all

ps -aef | grep snmp

all checks using check_snmp seems to be failing and returning error "CRITICAL - Plugin timed out while executing system call".

Is this to just one hosts or many different hosts?

[OptimusB]

This is happening to multiple hosts. I cannot get check_SNMP working. I am not sure what changed.

Looks like the normal for processes. Current SNMP checks.

Code: Select all

[root@XI libexec]#  ps -aef | grep snmp
nagios    2654 29485  0 10:21 ?        00:00:00 /usr/local/nagios/libexec/check_snmp -H Server -P 2c -c public -t 60 -o enterprises.789.1.6.4.7.0 -w 1 -c 2
nagios    2655  2654  0 10:21 ?        00:00:00 /usr/bin/snmpget -Le -t 60 -r 5 -m ALL -v 2c -c        server :161 enterprises.789.1.6.4.7.0
nagios    5388 29482  0 10:22 ?        00:00:00 /usr/local/nagios/libexec/check_snmp -H server2 -P 2c -c public -o enterprises.789.1.2.7.1.0 -c 1:1
nagios    5389  5388  0 10:22 ?        00:00:00 /usr/bin/snmpget -Le -t 1 -r 5 -m ALL -v 2c -c        server2:161 enterprises.789.1.2.7.1.0
root      6032  1961  0 10:22 pts/0    00:00:00 grep snmp

alerts.JPG

[tgriep]

It looks like the -c for the community string for the check_snmp is not in upper case, it should be like this -C
Try changing that and see if it fixes your issue.

[OptimusB]

I'll try, however my manual run of the SNMP check is used with a capital C and it is returning the same value. (result in first post)
Also this only happened a few days ago and it was working fine up to that point....

[Box293]

All of a sudden today, all checks using check_snmp seems to be failing and returning error "CRITICAL - Plugin timed out while executing system call". On some checks I am getting the error "(Service check timed out after 60.01 seconds)"
A browse through the forums mentioned a couple checks. So I've done the following:

1. Ran snmpwalk on the devices and it is returning proper values without issue.
2. Other snmp related check commands are working: ie. check_snmp_storage_wizard, check_snmp_load_wizard, etc.
3. ran check_snmp manually and it is also returning a timeout error

Code: Select all

./check_snmp -H server-P 2c -C public -o enterprises.789.1.5.4.1.6.1654 -t 30 -vvv
/usr/bin/snmpget -Le -t 30 -r 5 -m ALL -v 2c [authpriv] server:161 enterprises.789.1.5.4.1.6.1654
CRITICAL - Plugin timed out while executing system call

OK so the first line of the output with the -vvv mode turned on shows the snmpget command being used.

So let's run this command with the -D ALL option which enables a lot of debugging.
Also, I need to replace [authpriv] with -c public
And change "server" to the IP Address of the server you are checking against

Code: Select all

/usr/bin/snmpget -Le -t 30 -r 5 -m ALL -D ALL -v 2c -c public server:161 enterprises.789.1.5.4.1.6.1654

This should produce a lot more output.

[OptimusB]

Thanks for that command. So I was able to get a value from snmpget:

Code: Select all

NETAPP-MIB::dfPerCentKBytesCapacity.1654 = INTEGER: 75

However, I did run through the output and looked like there was a complain about a MIB that was recently added. I removed the MIB from /usr/share/snmp/mib and now check_snmp is working also.
For now I will keep the MIB out of there. Thanks!!!

[jdalrymple]

Good to lock and mark solved?

[OptimusB]

Yes. Thanks for the assistance!!