Nagios XI SNMPTT install script issues

[agriffin]

Please withdraw this patch if you released it completely breaks snmptrap handling. As snmptraphandling.py is never called so nagios never receives the processed traps !!!!!

What do you mean by patch, here? Do you mean the suggestions from your previous post or the package I recently put out?

[arnab.roy]

The previous RPM package, as well as any modifications.....the problem seems to be nothing is getting into nagios.cmd I added some debug to see if snmptraphandling.py is getting called or not and found it is not I couldnt find any reference to snmptraphandling.py in snmptthandler script so not sure how it writes the trap events into nagios....

[agriffin]

Okay, I would just like to verify some things. The RPM package currently doesn't do proper trap translation for you, nor does it pass traps to Nagios. The changes you mentioned earlier get trap translation to function, but they still don't integrate with Nagios. Is that correct?

Also, are you sure your custom OPTIONS line is even doing anything? If you have the snmptt package installed, that line should be getting overrided by /etc/sysconfig/snmptrapd or /etc/sysconfig/snmptrapd.options (depending on your distro version).

[arnab.roy]

Digging deeper if i look for output from snmptt it looks like this

karma-rbht snmptt[0]: healthMonDeviceTrap Normal "Status Events" 192.168.11.1 - Device-level rising/falling threshold crossing trap. noChange noChange noChange crossedBelowThreshold noChange noChange

As you can see its not printing the name of the service so I have a feeling some variable has got messed up ...somewhere ...will try back tracking all the changes and see what happens...

[arnab.roy]

OK re-installed Net-SNMP and your snmptt package which over wrote all my settings now I am back to a situtation where the trap oid is displayed in a numeric fashion and nothing is getting passed to Nagios...I am continuing to do some further tests and see whats going on...

[arnab.roy]

ok I think this is what is happening for some reason the formatting on the output from snmptt has gone off and I am getting this

usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

Agent IP address (172.20.0.14) is the same as the host IP, so copying the host name: wireless-loopback.khipu-networks.com

wlsxUserEntryAttributesChanged Normal "Status Events" wireless-loopback - 2012-10-30,23:30:13.0,-1:0 0.0.0.0 0:17:23:11:80:5c 0:1a:1e:86:13:a0 Khipu-Technical-Area 0 0 4
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Host IP address (2.0.1.42) could not be resolved by DNS. Variable $r / $R etc will use the IP address

Agent IP address was blank, so setting to the same as the host IP address of 2.0.1.42

Agent IP address (2.0.1.42) is the same as the host IP, so copying the host name: 2.0.1.42

wlsxAuthServerReqTimedOut Normal "Status Events" 2.0.1.42 - bns2
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

Agent IP address (172.20.0.14) is the same as the host IP, so copying the host name: wireless-loopback.khipu-networks.com

wlsxAPRadioEntryChanged Normal "Status Events" wireless-loopback - 2012-10-30,23:30:22.0,-1:0 21699 086:68:cc:5f 1 modify
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

Agent IP address (172.20.0.14) is the same as the host IP, so copying the host name: wireless-loopback.khipu-networks.com

wlsxAPChannelChange Normal "Status Events" wireless-loopback - 2012-10-30,23:30:22.0,-1:0 11 none 6 none 086:68:cc:5f 1 Test-WCC armErrorThresh
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

Agent IP address (172.20.0.14) is the same as the host IP, so copying the host name: wireless-loopback.khipu-networks.com

wlsxNRadioAttributesChanged Normal "Status Events" wireless-loopback - 2012-10-30,23:30:22.0,-1:0 086:68:cc:5f 0 10.10.10.40 11 40
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

Agent IP address (172.20.0.14) is the same as the host IP, so copying the host name: wireless-loopback.khipu-networks.com

wlsxAPRadioEntryChanged Normal "Status Events" wireless-loopback - 2012-10-30,23:30:26.0,-1:0 21700 0:24:6c:c2:2b:40 1 modify
usage: services.py <HOST> <SERVICE> <SEVERITY> <TIME> <PERFDATA> <DATA>
Agent IP address was blank, so setting to the same as the host IP address of 172.20.0.14

[arnab.roy]

Hi Guys...

Any idea whats going on ? I would like to go back to old style install of snmptt , would it be possible to get the old install script back ...

[arnab.roy]

For everyones benefit ...the trap handler into nagios is broken in this package please replace it with the attached and that should fix it

Note for Alex: Please upload this into the rpm to finally put this to grave / the previous modifications in snmptt.ini are important and should be done for proper trap translation ...

[Mt6uo]

What I've noticed is that the NagiosXI-SNMPTrap.sh (downloaded from here http://assets.nagios.com/downloads/nagi ... rap.tar.gz 20121031) is broken.

Code: Select all

# Install the SNMP trap translator
snmpttver=1.3
wget http://superb-sea2.dl.sourceforge.net/project/snmptt/snmptt/snmptt_${snmpttver}/snmptt_${snmpttver}.tgz
tar -xzf snmptt_${snmpttver}.tgz
cp ./snmptt_${snmpttver}/snmptt \
        ./snmptt_${snmpttver}/snmptthandler  /usr/local/sbin/
cp ./snmptt_${snmpttver}/snmptt.ini       /etc/snmp/
cp ./snmptt_${snmpttver}/snmptt-init.d    /etc/rc.d/init.d/snmptt/
cp ./snmptt_${snmpttver}/snmpttconvertmib /usr/local/bin/

sed -i 's_/usr/sbin_/usr/local/sbin_' /etc/rc.d/init.d/snmptt

/etc/rc.d/init.d/snmptt is both used a a file (last line with sed e.g.) and as a directory (cp ./snmptt_${snmpttver}/snmptt-init.d /etc/rc.d/init.d/snmptt/) ? I think this is what arnab.roy is experiencing here.

Ok just tried the new version and

Code: Select all

oaded plugins: fastestmirror, security
Determining fastest mirrors
epel/metalink                                                                                                                                             |  18 kB     00:00     
 * base: mirrors.coreix.net
 * epel: mirrors.coreix.net
 * extras: mirrors.coreix.net
 * rpmforge: www.mirrorservice.org
 * updates: mirrors.coreix.net
base                                                                                                                                                      | 3.7 kB     00:00     
cr                                                                                                                                                        | 3.0 kB     00:00     
epel                                                                                                                                                      | 4.3 kB     00:00     
epel/primary_db                                                                                                                                           | 4.7 MB     00:01     
extras                                                                                                                                                    | 3.0 kB     00:00     
rpmforge                                                                                                                                                  | 1.9 kB     00:00     
rpmforge/primary_db                                                                                                                                       | 2.5 MB     00:00     
updates                                                                                                                                                   | 3.5 kB     00:00     
updates/primary_db                                                                                                                                        | 3.4 MB     00:01     
vmware-tools                                                                                                                                              |  951 B     00:00     
Setting up Install Process
Package 1:net-snmp-5.5-41.el6_3.1.x86_64 already installed and latest version
Package rpmdevtools-7.5-1.el6.noarch already installed and latest version
Package 1:net-snmp-perl-5.5-41.el6_3.1.x86_64 already installed and latest version
Package perl-Config-IniFiles-2.72-2.el6.noarch already installed and latest version
Nothing to do
--2012-10-18 20:03:13--  http://superb-sea2.dl.sourceforge.net/project/snmptt/snmptt/snmptt_1.3/snmptt_1.3.tgz
Resolving superb-sea2.dl.sourceforge.net... 209.160.57.180
Connecting to superb-sea2.dl.sourceforge.net|209.160.57.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 146314 (143K) [application/octet-stream]
Saving to: `snmptt_1.3.tgz'

100%[=======================================================================================================================================>] 146,314      157K/s   in 0.9s    

2012-10-18 20:03:14 (157 KB/s) - `snmptt_1.3.tgz' saved [146314/146314]

cp: cannot create regular file `/etc/rc.d/init.d/snmptt/': Is a directory

Mind you this system already had the previous installer run on it....

Can you guys kindly test this please....

The fix should be to remove the / at the end of /etc/rc.d/init.d/snmptt to signify it's not a directory.

Code: Select all

# Install the SNMP trap translator
snmpttver=1.3
wget http://superb-sea2.dl.sourceforge.net/project/snmptt/snmptt/snmptt_${snmpttver}/snmptt_${snmpttver}.tgz
tar -xzf snmptt_${snmpttver}.tgz
cp ./snmptt_${snmpttver}/snmptt \
        ./snmptt_${snmpttver}/snmptthandler  /usr/local/sbin/
cp ./snmptt_${snmpttver}/snmptt.ini       /etc/snmp/
cp ./snmptt_${snmpttver}/snmptt-init.d    /etc/rc.d/init.d/snmptt
cp ./snmptt_${snmpttver}/snmpttconvertmib /usr/local/bin/

sed -i 's_/usr/sbin_/usr/local/sbin_' /etc/rc.d/init.d/snmptt

[Mt6uo]

I've troubleshooted some more now:

Trap entering on interface:

Code: Select all

# tcpdump -i eth1 -n udp port 162
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
14:23:40.419449 IP 10.254.30.12.32789 > 10.232.130.65.snmptrap:  C=obfuscated V2Trap(201)  .1.3.6.1.2.1.1.3.0=1881160600 .1.3.6.1.6.3.1.1.4.1.0=.1.3.6.1.4.1.14823.2.3.1.11.1.2.1224 .1.3.6.1.4.1.14823.2.3.1.11.1.1.60=07_dc_0b_01_0f_0c_14_00_2b_01_00 .1.3.6.1.4.1.14823.2.3.1.11.1.1.53.0="fakeloginname" .1.3.6.1.4.1.14823.2.3.1.11.1.1.51.0=10.232.120.1 .1.3.6.1.4.1.14823.2.3.1.11.1.1.26.0="obfuscated"

A reverse lookup on the source IP of the trap from Nagios:

Code: Select all

;; QUESTION SECTION:
;12.30.254.10.in-addr.arpa.     IN      PTR

;; ANSWER SECTION:
12.30.254.10.in-addr.arpa. 3600 IN      PTR     hostname.zonename.

/var/log/snmptt/snmptt.log file:

Code: Select all

Thu Nov  1 14:23:40 2012 .1.3.6.1.4.1.14823.2.3.1.11.1.2.1224 Normal "Status Events" hostname -   07 DC 0B 01 0F 0C 14 00 2B 01 00  fakeloginname obfuscated-ip obfuscated-fqdn

I hooked the snmptraphandling.py script to print the sys.argv variable received from snmptt:

Code: Select all

/usr/local/bin/snmptraphandling.py "hostname" "SNMP Traps" "Normal" "1351767722" "enterprises.14823.2.3.1.11.1.1.60 ():07 DC 0B 01 0C 32 28 00 2B 01 00  enterprises.14823.2.3.1.11.1.1.53.0 ():fakeloginname enterprises.14823.2.3.1.11.1.1.51.0 ():10.232.120.1 enterprises.14823.2.3.1.11.1.1.26.0 ():obfuscated-fqdn" "                        07 DC 0B 01 0C 32 28 00 2B 01 00  fakeloginname  obfuscated-ip obfuscated-fqdn"

When I ran the command on the CLI, writes to /usr/local/nagios/var/rw/nagios.cmd without problem. However, no trap appears in the trap service of the host. But when I change the first argument to the script to contain the FQDN instead of just the hostname:

Code: Select all

/usr/local/bin/snmptraphandling.py [b]"FQDN"[/b] "SNMP Traps" "Normal" "1351767722" "enterprises.14823.2.3.1.11.1.1.60 ():07 DC 0B 01 0C 32 28 00 2B 01 00  enterprises.14823.2.3.1.11.1.1.53.0 ():fakeloginname enterprises.14823.2.3.1.11.1.1.51.0 ():10.232.120.1 enterprises.14823.2.3.1.11.1.1.26.0 ():obfuscated-fqdn" "                        07 DC 0B 01 0C 32 28 00 2B 01 00  fakeloginname  obfuscated-ip obfuscated-fqdn"

It works. The trap appears to my host in Nagios. However, it's not translated. I have translated the appropriate MIB into /etc/snmp/snmptt.conf:

Code: Select all

[...]
EVENT wlsxNUserAuthenticationFailed .1.3.6.1.4.1.14823.2.3.1.11.1.2.1018 "Status Events" Normal
FORMAT                   $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "                      $*"
SDESC

This trap indicates that a user authentication has failed.

Variables:
  1: wlsxTrapTime
  2: wlsxTrapUserIpAddress
  3: wlsxTrapUserPhyAddress
EDESC
[...]

Then I noticed this directive in /etc/snmp/snmptt.ini:

Code: Select all

strip_domain = 1

Changed it to 0. Now traps are shown in Nagios for my hosts. However, they are not translated. My snmptt.ini is configured according to the previous posts in this thread. Any suggestions as how to troubleshoot translations would be welcome.

Edit: I've tried with traps from other vendors and they are translated just fine. I think I've been provided the wrong MIB for the Aruba system I initially tried to traps from.