Linux SNMP: Process name table No response from remote host

[msbensonstk]

I am having trouble running SNMP v3 checks against some RedHat Linux hosts.

The check plugin being run is check_snmp_process_wizard.pl with the -f flag (as the process I need to find can only be found using the fullpath option)

In the Nagios GUI, the error message I am getting back from the check is ERROR: Process name table : No response from remote host

Running the check from the command line of the Nagios server (the date commands are there for checking the timing of the command):
date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <omitted> --login=<omitted> --passwd=<omitted> --privpass=<omitted> --protocols=sha,aes -n 'ora_pmon_*' -f -w '0,1' -c '0,1'; date
Fri Jun 21 08:49:57 CDT 2013
ERROR: Process name table : No response from remote host '<omitted>'.
Fri Jun 21 08:50:07 CDT 2013

However if I omit the -f flag I get a response (but I need the -f to find the process I am looking for - this is just to demonstrate that SNMP works):
date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <omitted> --login=<omitted> --passwd=<omitted> --privpass=<omitted> --protocols=sha,aes -n 'ora_pmon_*' -w '0,1' -c '0,1'; date
Fri Jun 21 09:07:56 CDT 2013
No process matching ora_pmon_* found : CRITICAL
Fri Jun 21 09:07:57 CDT 2013

Now, I know this is not a SNMP user permission problem because I can execute an snmpwalk of the host and find the process I am looking for with no problem:
snmpwalk -v 3 -l authPriv -a sha -A <omitted> -x aes -X <omitted> -u <omitted> "<omitted>" | grep ora_pmon_*
HOST-RESOURCES-MIB::hrSWRunPath.4562 = STRING: "ora_pmon_GRID"

Thinking this may be a timeout problem trying to check the path table rather than the process name table, I tried adding the --timeout flag to the command:
date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <omitted> --login=<omitted> --passwd=<omitted> --privpass=<omitted> --protocols=sha,aes -n 'ora_pmon_*' -f --timeout=10 -w '0,1' -c '0,1'; date
Fri Jun 21 09:12:50 CDT 2013
ERROR: Alarm signal (Nagios time-out)
Fri Jun 21 09:13:06 CDT 2013

With the timeout flag, I get a different error, but looking at the 'date' command output, it looks like the command is ignoring my custom timeout value: Fri Jun 21 09:12:50 CDT 2013 -> Fri Jun 21 09:13:06 CDT 2013 = 15 seconds - timeout was set to 10.

I tried this with a very small timeout value and got the same result:
date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <omitted> --login=<omitted> --passwd=<omitted> --privpass=<omitted> --protocols=sha,aes -n 'ora_pmon_*' -f --timeout=5 -w '0,1' -c '0,1'; date
Fri Jun 21 09:14:43 CDT 2013
ERROR: Process name table : No response from remote host '<omitted>'.
Fri Jun 21 09:14:54 CDT 2013
Check duration: Fri Jun 21 09:14:43 CDT 2013 -> Fri Jun 21 09:14:54 CDT 2013 = 11 seconds instead of 5 seconds.

Thinking this may be using one of the configured nagios timeouts I checked all of the .cfg files:
grep timeout= /usr/local/nagios/etc/*
/usr/local/nagios/etc/nagios.cfg:event_handler_timeout=30
/usr/local/nagios/etc/nagios.cfg:host_check_timeout=30
/usr/local/nagios/etc/nagios.cfg:notification_timeout=30
/usr/local/nagios/etc/nagios.cfg:ocsp_timeout=5
/usr/local/nagios/etc/nagios.cfg:perfdata_timeout=5
/usr/local/nagios/etc/nagios.cfg:service_check_timeout=60
/usr/local/nagios/etc/ndomod.cfg:file_rotation_timeout=60
/usr/local/nagios/etc/nrpe.cfg:command_timeout=60
/usr/local/nagios/etc/nrpe.cfg:connection_timeout=300

None of the timeout values are set to 15 seconds. So, I have no idea why:
1. The -f flag is causing these commands to fail
2. Why the command is not obeying the --timeout flag

Any help would be appreciated

[sreinhardt]

1) F is likely to fail just looking at your command, because it requires the full path. It seems that you are only providing ora_pmon_.
2) It appears that the script uses two timeout variables. A $TIMEOUT=15 and $o_timeout=5 or user set time. It does send the o_timeout to net::snmp plugin, so this should be respected. If you set it to something more like 30 seconds, do you notice it continuing longer?

[msbensonstk]

Sorry I didn't get back on this sooner, but the reply got eaten by my spam folder.

1. F works in this method as I am using -n which allows for regex matching (ora_pmon_*) - I have tested this successfully with other systems and can supply output of the success if needed for verification.

2. o_timeout=0; TIMEOUT=60; date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <host address omitted> --login=<username> --passwd=<password> --privpass=<password> --protocols=sha,aes -n 'ora_pmon_*' -f -w '0,1' -c '0,1'; date
Thu Jul 18 16:51:50 CDT 2013
ERROR: Process name table : No response from remote host '<host address omitted>'.
Thu Jul 18 16:52:00 CDT 2013

o_timeout=0; TIMEOUT=0; date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H <host address omitted> --login=<username> --passwd=<password> --privpass=<password> --protocols=sha,aes -n 'ora_pmon_*' -f --timeout=5 -w '0,1' -c '0,1'; date
Thu Jul 18 16:54:10 CDT 2013
ERROR: Process name table : No response from remote host '<host address omitted>'.
Thu Jul 18 16:54:20 CDT 2013

The process still runs for 10 seconds regardless of what I enter for a --timeout value or what I set $o_timeout and $TIMEOUT to.

[sreinhardt]

I see this floating around on a few other forums as well, although generally older posts. Let's go ahead and do a quick snmpwalk with some timing differences to see what happens.

snmpwalk -v 3 -a sha -x AES -u [username] -A [password] -X [privacy passphrase] -t 10 [Host IP] 1.3.6.1.2.1.25.4.2.1 | grep -i 'ora_pmon_' > /tmp/walk-10
snmpwalk -v 3 -a sha -x AES -u [username] -A [password] -X [privacy passphrase] -t 30 [Host IP] 1.3.6.1.2.1.25.4.2.1 | grep -i 'ora_pmon_' > /tmp/walk-30
snmpwalk -v 3 -a sha -x AES -u [username] -A [password] -X [privacy passphrase] -t 60 [Host IP] 1.3.6.1.2.1.25.4.2.1 | grep -i 'ora_pmon_' > /tmp/walk-60
snmpwalk -v 3 -a sha -x AES -u [username] -A [password] -X [privacy passphrase] -t 90 [Host IP] 1.3.6.1.2.1.25.4.2.1 | grep -i 'ora_pmon_' > /tmp/walk-90
tar cjf walks.tar.bz2 /tmp/walk-*

Please send back that walk tar file. Additionally, are you using the host IP or name, I see address listed in your examples, but some people do confused them.

[msbensonstk]

I had to change the command slightly as there was a missing parameter and the system I picked had no ora_pmon processes on it. The command I ran was:

for timeout in 10 30 60 90;
do snmpwalk -v 3 -l authPriv -a sha -A [password] -x aes -X [password] -u [user] -t $timeout [IP Address] | grep -i '/usr/local/contego/ContegoSPOP/' > /tmp/walk-$timeout;
done;
tar cjf walks.tar.bz2 /tmp/walk-*

All of the walks succeeded and returned results (as indicated in the walks.tar.bz2).

This is what I get back using the check_snmp_process_wizzard.pl checking for the same process on the same system:

date; /usr/local/nagios/libexec/check_snmp_process_wizard.pl -H [IP Address] --login=[user] --passwd=[password] --privpass=[password] --protocols=sha,aes -n '/usr/local/contego/ContegoSPOP/' -f --timeout=5 -w '0,1' -c '0,1'; date
Mon Jul 22 10:44:14 CDT 2013
ERROR: Process name table : No response from remote host '[IP Address]'.
Mon Jul 22 10:44:25 CDT 2013

[sreinhardt]

Wow, ok so you absolutely can get snmp working and that oid is just what you are expecting. Could you run a tcpdump of the nagios plugin trying to execute a snmpget and post it? If you are worried about interal IPs, feel free to pm me instead.

[msbensonstk]

What flags (i.e. how much hex code) do you want in the dump?

[sreinhardt]

A raw pcap\cap would be best. I would like to see everything on ports 161 162. Something like:

Code: Select all

tcpdump -i eth0 -K -nn -w /tmp/snmp.pcap portrange 161-162

This should allow it to capture "bad" packets, not resolve port or hostname, write to the temp file /tmp/snmp.pcap, and specify any src\dst ports 161 or 162. Then if you could run the check_snmp_process_wizard.pl plugin as you normally would.

[msbensonstk]

PM'd pcap file.

[sreinhardt]

Got it, I will take a look!