SNMP trap config

[gormank]

Hi,
After configuring things some time ago, I'm seeing lots of traps in /var/log/messages. I'll create a test trap to make this specific to one host. These are ILOs, 3PAR SAN, fiber switches, etc., not servers.
If I go into the ilo and create a test trap, there it is:

Code: Select all

Sep 16 18:02:26 txslm2mlnag001 snmptrapd[1464]: 2015-09-16 18:02:26 10.133.133.23(via UDP: [10.133.133.23]:59232->[10.133.133.74]) TRAP, SNMP v1, community sp1der#012#011.1.3.6.1.4.1.232 Enterprise Specific Trap (11003) Uptime: 415 days, 5:54:40.00#012#011.1.3.6.1.2.1.1.5.0 = STRING: txslm2mlapp002-ilo.#011.1.3.6.1.4.1.232.11.2.11.1.0 = INTEGER: 4#011.1.3.6.1.4.1.232.11.2.8.1.0 = STRING: "Remote Insight Test Trap"

Sep 16 18:02:56 txslm2mlnag001 snmptrapd[1464]: 2015-09-16 18:02:56 10.133.133.23(via UDP: [10.133.134.96]:50005->[10.133.134.84]) TRAP, SNMP v1, community sp1der#012#011.1.3.6.1.4.1.232 Enterprise Specific Trap (11003) Uptime: 0:00:00.00#012#011.1.3.6.1.2.1.1.5.0 = STRING: txslm2mlapp002-ilo.#011.1.3.6.1.4.1.232.11.2.11.1.0 = INTEGER: 4#011.1.3.6.1.4.1.232.11.2.8.1.0 = STRING: "Remote Insight Test Trap"

But the traps don't show on my hosts. For example, the trap above from 10.133.133.23 is from txslm2mlapp002-ilo, which is an ILO.

Code: Select all

# grep 10.133.133.23 *
txslm2mlapp002-ilo.cfg: address                         10.133.133.23
[root@txslm2mlnag001 hosts]# pwd
/usr/local/nagios/etc/hosts

I disabled the service I'd adapted from the trap wizard and created a new one with the wizard, renamed it and applied my SNMP hostgroup to it.

Code: Select all

# grep -v ^# ../services/SNMP_Trap_Wizard_test.cfg

define service {
        host_name                       txslm2m3prf001,txslm2mlnag001
        service_description             SNMP_Trap_Wizard_test
        use                             xiwizard_snmptrap_service
        hostgroup_name                  1VZW_SNMP_trap_hosts
        max_check_attempts              1
        check_interval                  1
        retry_interval                  1
        check_period                    xi_timeperiod_24x7
        notification_interval           1
        notification_period             xi_timeperiod_24x7
        contacts                        nagiosadmin
        icon_image                      snmptrap.png
        _xiwizard                       snmp_trap
        register                        1
        }

The xiwizard_snmptrap_service template:

Code: Select all

define service {
       name                                     xiwizard_snmptrap_service
       service_description                      SNMP Traps
       use                                      xiwizard_generic_service
       check_command                            check_dummy!0!"TRAP RESET"!!!!!!
       is_volatile                              1
       initial_state                            o
       active_checks_enabled                    0
       passive_checks_enabled                   1
       flap_detection_enabled                   0
       stalking_options                         o,w,c,u,
       icon_image                               snmptrap.png
       register                                 0
}

The SNMP_Trap_Wizard_test service stays in pending state until run manually, then it goes green due to check_dummy returning 0. Apparently there's some magic in setting the intervals all to 1 to make it never run the actual check script and just wait for traps. This is good.

The traps aren't in the unknown log or unknown traps in Nagios.

Code: Select all

# ll /var/log/snmptt/snmpt*log
-rw-rw-r-- 1 snmptt root   0 Sep 13 03:31 /var/log/snmptt/snmptt.log
-rw-rw-r-- 1 snmptt snmptt 0 Sep  6 03:08 /var/log/snmptt/snmpttsystem.log
-rw-rw-r-- 1 snmptt root   0 Sep 13 03:31 /var/log/snmptt/snmpttunknown.log

The question is, why don't the traps end up on the host?

[ssax]

Please post your /etc/snmp/snmptrapd.conf, if you're not seeing them come into snmptt that config is the likely problem.

The process is as follows:

Device -> XI Server -> snmptrapd -> snmptt -> /usr/local/bin/snmptraphandling.py -> Nagios

[gormank]

Looks ok to me...

Code: Select all

# cat /etc/snmp/snmptrapd.conf
disableAuthorization yes
traphandle default /usr/local/sbin/snmptthandler

# ll /usr/local/sbin/snmptthandler
-rwxr-xr-x 1 root root 6488 Nov  1  2012 /usr/local/sbin/snmptthandler

After messing with the SNMP services in Nagios and/or restarting snmptt and snmptrapd, there are devices in unconfigured objects. These hosts actually exist as hosts in Nagios so I'm not sure why...

[ssax]

The names need to be identical, it's based on the host_name, not the address. Are they identical (case sensitive, with or without domain name, or IP instead of dns name)?

[gormank]

We can see from my example above that the 10.133.133.23 host has the name txslm2mlapp002-ilo. You can see that name in the trap examples. So the hostname in the ILO and in Nagios is identical. I actually changed the ILO hostname before posting first to make sure that wasn't the problem.

In DNS, the name is txslm2mlapp002.ilo

The odd thing is, this all seems to have been working up until 9/9, a week ago. This makes me think the issue is on the hosts, or more likely, templates. /var/log/snmptt/snmptt.log has lots of entries up until that date...

[ssax]

Oh, sorry, I think it's because you changed the service description, take a look at this example trap from /etc/snmp/snmptt.conf:

Code: Select all

EVENT nCustomEvent .1.3.6.1.4.1.20006.1.99 "Status Events" Critical
FORMAT $*
EXEC /usr/local/bin/snmptraphandling.py "$1" "SNMP Traps" "$s" "$@" "" "$2"
SDESC
The SNMP trap that is generated as a result of an event with the service
in Nagios.
Variables:
  1: nHostname
  2: nSvcOutput
EDESC

The EXEC line is what throws the trap into Nagios, the "SNMP Traps" part is the service description, that needs to match.

[gormank]

There's a forehead slapper...
I never even thought about the description making a difference.
I changed it back so I'll watch things and post results.

Thanks!

[gormank]

Strange. After the change, I started seeing lots of traps in Nagios, but they seemed to be old based on the duration dates. So I disabled the service and enabled it to clear all the messages. After that everything's back to "No check results for service yet..."

Traps are in messages, but not getting into Nagios.
There are no new entries in the smptt*log files.

[gormank]

This is crazy. Just for grins I thought I'd restart snmptt and snmptrapd again, and as soon as I did, the traps reappeared in Nagios.
I'll have another look tomorow. snmptt usually shows the shutdown failed when restarting.

# service snmptt restart
Stopping snmptt: [FAILED]
Starting snmptt: [ OK ]

[ssax]

Are you seeing anything errors or anything in your /var/log/messages or /var/log/snmptt/*?

Please attach your /etc/snmp/snmptt.ini so that I can review its settings.

Thank you