Plugin time out error for islion box

[udaykumar]

Hi team,

All my isilion boxes are being monitored by using check_snmp (/check_snmp -H $HOSTADDRESS$ $ARG1$)

From few days we are facing the issue as plugin time out in nagios console.

Attaching screen shot for reference. Please help us to resolve the issue asap.

[lmiltchev]

Is the server that you are monitoring very busy? Anything changed - an IP, community string, etc.?

Can you show us one of the failing checks, run from the command line along with the output of it? Try timing the check:

Code: Select all

time /usr/local/nagios/libexec/check_snmp <rest of your command>

What is the global timeout for services in the nagios.cfg set at? Run the command below and show the output:

Code: Select all

grep service_check_timeout= /usr/local/nagios/etc/nagios.cfg

[udaykumar]

Hi Team,

We dint change any IP or string.

For first command output is ,


real 0m11.004s
user 0m0.000s
sys 0m0.003s

CRITICAL - Plugin timed out while executing system call

And second command output is
service_check_timeout=60

Request to review asap.

[lmiltchev]

What is the load on the remote machine? Is snmpd running on the islion box? Is there a firewall in between? Is UDP port 161 open?

Code: Select all

nmap -sU <remote ip> -p 161

Your check timed out after 10 seconds as this is the "default" timeout for check_snmp plugin. Try increasing the timeout by using the "-t" flag.

Code: Select all

/usr/local/nagios/libexec/check_snmp <rest of your command> -t 50

Did the check succeed?

[udaykumar]

Hello,

Port is opened and snmpd is running.

nmap result is .

Starting Nmap 6.47 ( http://nmap.org ) at 2018-11-28 09:37 AEDT
Nmap scan report for xxxxxxxx (xxxxxxxx)
Host is up (0.00038s latency).
PORT STATE SERVICE
161/udp open|filtered snmp



And even after increasing tme out to -t 50, result is same.

CRITICAL - Plugin timed out while executing system call

real 0m51.004s
user 0m0.001s
sys 0m0.003s

[lmiltchev]

I never fully trust the nmap's udp scans... Having "filtered" in the output:

161/udp open|filtered snmp

which may (or may not) indicate a firewall issue. The only way to be 100% sure, is to temporarily stop the firewall, and try running your check from the CLI.

FYI, when I run nmap against my test VM, I get:

161/udp open snmp

Since these are SNMP checks, let's try one more troubleshooting step - run the snmpwalk from the command line on the Nagios XI server:

Code: Select all

snmpwalk -v2c -c <community string> <remote ip address>

Do you get some output or the command times out?

If nagios cannot access the remote machine due to SNMP service problem or firewall issues, your checks won't work.

[udaykumar]

HI Team,

Please find the response of command.

[root@xxxxxxx libexec]# snmpwalk -v 2c -c Nagios 10.xxxxxxxxxxxx
Timeout: No Response from 10.xxxxxxxxx

[lmiltchev]

If you are not able to perform snmpwalk against the remote machine, your check won't work...

Have you tested your check with firewall turned off?

Can you run snmpwalk on the remote machine (locally)?

Code: Select all

snmpwalk -v 2c -c Nagios localhost

Have you made any changes to the /etc/snmp/snmpd.conf file prior to seeing this issue? Please post the snmpd.conf file on the forum.

[udaykumar]

Hi,

We are not using any fire wall between nagios server and remote host.
We have run the below command in remote machine and below s out put.

Code: Select all

root-1# snmpwalk -v 2c -c I\$ilonpublic localhost
SNMPv2-MIB::sysDescr.0 = STRING: acpnpcisi01-1 3519032624 Isilon OneFS v8.0.0.4
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.12325.1.1.2.1.1
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (2932009613) 339 days, 8:28:16.13
SNMPv2-MIB::sysContact.0 = STRING: [email protected]
SNMPv2-MIB::sysName.0 = STRING: acpnpcisi01-1
SNMPv2-MIB::sysLocation.0 = STRING: unset
SNMPv2-MIB::sysServices.0 = INTEGER: 76
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (8) 0:00:00.08 

And we dint change anything in snmpd file. Attaching file below.

[lmiltchev]

And we dint change anything in snmpd file. Attaching file below.

Are you sure this is the correct config (from the remote server)? You community string is set to the default (public)...

Code: Select all

com2sec notConfigUser  default       public

You've been using two other community strings - "Nagios" and "I\$ilonpublic"... So, which one is the correct one? We need to clarify this before we can even begin troubleshooting this issue!

Also, I see a very little output from the snmpwalk that was run "locally". If this is indeed your config, comment out these two lines:

Code: Select all

view    systemview    included   .1.3.6.1.2.1.1
view    systemview    included   .1.3.6.1.2.1.25.1.1

and add another line below, including the entire tree:

Code: Select all

# view    systemview    included   .1.3.6.1.2.1.1
# view    systemview    included   .1.3.6.1.2.1.25.1.1
view    systemview    included   .1

Save, exit, and restart snmpd:

Code: Select all

service snmpd restart