Nagios Support Forum

Version:
We have: NagiosXi 2011 R1.6

Problem:
I ran the SNMP Trap Wizard and pointed it to the corresponding server. The server sending the snmp traps sends the traps, are acknowledge by the Nagios CentOS server in the logs, but NagiosXi still shows "Waiting for Traps".

Steps Taken
I've followed the instructions on: http://assets.nagios.com/downloads/nagi ... ith_XI.pdf

Logs
Below is a copy of the /var/log/messages (I modified the server names on the log below to maintain security):

Aug 5 10:20:05 "Our Nagios Server" snmptrapd[13249]: 2011-08-05 10:20:05 "ServerSendingTheSNMPvia SiteMinder" [UDP: [89.0.2.63]:2640]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 5 10:20:07 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "ServerSendingTheSNMPvia SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 5 10:20:08 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"ServerSendingTheSNMPvia SiteMinder"', but the host could not be found!

Aug 5 10:20:14 "Our Nagios Server" snmptrapd[13249]: 2011-08-05 10:20:14 "ServerSendingTheSNMPvia SiteMinder" [UDP: [89.0.2.63]:2640]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 5 10:20:14 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "ServerSendingTheSNMPvia SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 5 10:20:15 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"ServerSendingTheSNMPvia SiteMinder"', but the host could not be found!

Aug 5 10:20:15 "Our Nagios Server" snmptrapd[13249]: 2011-08-05 10:20:15 "ServerSendingTheSNMPvia SiteMinder" [UDP: [89.0.2.63]:2640]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 5 10:20:15 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "ServerSendingTheSNMPvia SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 5 10:20:16 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"ServerSendingTheSNMPvia SiteMinder"', but the host could not be found!

Aug 5 10:20:23 "Our Nagios Server" snmptrapd[13249]: 2011-08-05 10:20:23 "ServerSendingTheSNMPvia SiteMinder" [UDP: [89.0.2.63]:2640]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 5 10:20:23 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "ServerSendingTheSNMPvia SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 5 10:20:23 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"ServerSendingTheSNMPvia SiteMinder"', but the host could not be found!

Aug 5 10:21:05 "Our Nagios Server" xinetd[2525]: START: nrpe pid=7893 from="IP address of the Nagios Server"

Aug 5 10:21:05 "Our Nagios Server" xinetd[2525]: START: nrpe pid=7895 from="IP address of the Nagios Server"

Aug 5 10:21:06 "Our Nagios Server" xinetd[2525]: EXIT: nrpe status=0 pid=7893 duration=1(sec)

Aug 5 10:21:07 "Our Nagios Server" xinetd[2525]: EXIT: nrpe status=0 pid=7895 duration=2(sec)

Aug 5 10:21:52 "Our Nagios Server" xinetd[2525]: START: nrpe pid=8249 from="IP address of the Nagios Server"

Aug 5 10:21:52 "Our Nagios Server" xinetd[2525]: START: nrpe pid=8254 from="IP address of the Nagios Server"

Aug 5 10:21:52 "Our Nagios Server" auditd[2048]: Audit daemon rotating log files

Aug 5 10:21:52 "Our Nagios Server" xinetd[2525]: EXIT: nrpe status=0 pid=8249 duration=0(sec)

Aug 5 10:21:58 "Our Nagios Server" snmptrapd[13249]: 2011-08-05 10:21:58 "ServerSendingTheSNMPvia SiteMinder" [UDP: [89.0.2.63]:2640]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 5 10:21:58 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "ServerSendingTheSNMPvia SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 5 10:21:58 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"ServerSendingTheSNMPvia SiteMinder"', but the host could not be found!

Is there anything showing up under Admin > Unconfigured Objects?

I've attached the "Unconfigured Objects" screenshot.

I"ve read through the entire "Nagios Xi - Integrating SNMP Traps" followed line by line, but still can't get this to work. Please help.

What we are trying to accomplish with the SNMP Traps is capture data such, as AUTH Failures from Siteminder.

Thanks

Install Unconfigured Objects wizard as noted on the Unconfigured Objects page, a doc on how to use that wizard is here:

http://assets.nagios.com/downloads/nagi ... ith_XI.pdf

When thats finally setup, that should start piping those traps as per the setup for the unconfigured object.

I resolved the "Unconfigured Objects", but it is still not finding the Host:

This is the latest /var/log/messages log:

Aug 8 08:47:50 "Our Nagios Server" snmptrapd[13249]: 2011-08-08 08:47:50 "Server Sending the SNMP via SiteMinder" [UDP: [89.0.2.63]:3717]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 8 08:47:50 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "Server Sending the SNMP via SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 8 08:47:50 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"Server Sending the SNMP via SiteMinder"', but the host could not be found!

Aug 8 08:47:52 "Our Nagios Server" snmptrapd[13249]: 2011-08-08 08:47:52 "Server Sending the SNMP via SiteMinder" [UDP: [89.0.2.63]:3717]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 8 08:47:52 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "Server Sending the SNMP via SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 8 08:47:52 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"Server Sending the SNMP via SiteMinder"', but the host could not be found!

Aug 8 08:47:54 "Our Nagios Server" snmptrapd[13249]: 2011-08-08 08:47:54 "Server Sending the SNMP via SiteMinder" [UDP: [89.0.2.63]:3717]: .1.3.6.1.2.1.1.3.0 = Timeticks: (0) 0:00:00.00 .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.6.3.1.1.5.1

Aug 8 08:47:55 "Our Nagios Server" snmptt[0]: .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" "Server Sending the SNMP via SiteMinder" - A coldStart trap signifies that the SNMP entity,

Aug 8 08:47:55 "Our Nagios Server" nagios: Warning: Passive check result was received for service 'SNMP Traps' on host '"Server Sending the SNMP via SiteMinder"', but the host could not be found!

ecarrasq,

I'm currently working on restructuring the doc on this, but I'll need your help for some real situation experience on it.

First make a snmptt directory in the /var/log directory

mkdir /var/log/snmptt

Now edit /etc/snmp/snmptt.ini , and set the variables to the following,
you'll have to find them and change their values;

mibs_environment = ALL #simply uncomment this one
keep_unlogged_traps = 1
stdout_enable = 1
log_enable = 1
log_file = /var/log/snmptt/snmptt.log
log_system_enable = 1
log_system_file = /var/log/snmptt/snmpttsystem.log
unknown_trap_log_enable = 1
unknown_trap_log_file = /var/log/snmptt/snmpttunknown.log

Okay, now restart snmptt

service snmptt restart

Verify that our above steps worked:

ls /var/log/snmptt

This should return something, if it doesn't, make sure you found the
correct variables.

Now, what I think is causing the issue here is that the snmptt service
is not handing the trap off to Nagios XI, due to the fact that it
doesn't know what it being undefined, so lets that that theory.

when you do a:

tail -f /var/log/snmptt/snmptt.log

You should see a real-time log of what snmp traps you're receiving, and
you've verified that you're receiving them. However your logs indicate
that some of the logs are coming as unknown. In order to remedy that,
wait for one to come in, then check to see if an snmpttunknown.log

tail -f /var/log/snmptt/snmpttunknown.log

When one of these comes in, it will have the structure like this (this
is the example I reproduced on my machine):

>Mon Aug 8 17:38:53 2011: Unknown trap (.1.3.6.1.4.1.2021.13.990.0.17)
>received from 192.168.5.88 at:
>Value 0: 192.168.5.88
>Value 1: 192.168.5.88
>Value 2: 0:0:09:02.76
>Value 3: .1.3.6.1.4.1.2021.13.990.0.17
>Value 4: 192.168.5.88
>Value 5: public
>Value 6: .1.3.6.1.4.1.2021.13.990
>Value 7:
>Value 8:
>Value 9:
>Value 10:
>Ent Value 0: .1.3.6.1.2.1.1.6.0=Old Ben Kenobi

The part of that message that is very important here is this part:

Mon Aug 8 17:38:53 2011: Unknown trap (.1.3.6.1.4.1.2021.13.990.0.17)

The OID in here is what we're going to use.

Now its time to edit your /etc/snmp/snmptt.conf, using the OID that was
after the Unknown trap part (mine was .1.3.6.1.4.1.2021.13.990.0.17),
we're going to insert this block of text into snmptt.conf . Replace the
<< PUT THE OID ... >> with the OID from your unknown SNMP trap.

EVENT sysLocation << PUT THE OID FROM THE UNKNOWN RIGHT HERE >> "Status
Events" Normal
FORMAT "Syslocation: $1"
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@"
"$-*" "Notification that the trigger indicated by the object $*"

This can be inserted after any of the pound signs. Now save and exit and
restart snmptt

service snmptt restart

Now if you have a way of generating the same snmp trap again thats ideal
to test it out.

Now if this was indeed your problem, you will need to import all your
MIBs again because thats the root of this issue. However, let me know
how this goes as I'm very interested in the result.

I'm sorry that this can't be any easier.

I tried what you suggested, but it is still not working.

After doing the:

1. mkdir /var/log/snmptt

2. Now edit /etc/snmp/snmptt.ini

mibs_environment = ALL #simply uncomment this one
keep_unlogged_traps = 1
stdout_enable = 1
log_enable = 1
log_file = /var/log/snmptt/snmptt.log
log_system_enable = 1
log_system_file = /var/log/snmptt/snmpttsystem.log
unknown_trap_log_enable = 1
unknown_trap_log_file = /var/log/snmptt/snmpttunknown.log


3. service snmptt restart

service snmptt restart
Stopping snmptt: [ OK ]
Starting snmptt: MIB search path: $HOME/.snmp/mibs:/usr/local/share/snmp/mibs
Cannot find module (UCDSNMPMIB): At line 2 in /usr/local/share/snmp/mibs/TRAP-TEST-MIB.txt
[ OK ]


4. ls /var/log/snmptt
snmptt.log snmpttsystem.log



5. While traps were being sent, I never received the "snmpttunknown.log"

I really think you are correct about, "Now, what I think is causing the issue here is that the snmptt service
is not handing the trap off to Nagios XI."

Can you verify that you are receiving them by checking the smptt.log?

Yes:

snmptt]# tail -f snmptt.log
Wed Aug 10 13:19:18 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 13:19:19 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 13:19:20 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 13:19:21 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 13:19:21 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 13:19:22 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 15:43:01 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,
Wed Aug 10 15:45:56 2011 .1.3.6.1.6.3.1.1.5.1 Normal "Status Events" << Server Sending The Traps >> - A coldStart trap signifies that the SNMP entity,

I also verified and here is more information on how the traps are being sent. The tool they are using is "iReasoning MIB Browser"
1. The MIB is loaded
2. They scroll down to an event and right click on it
3. Select "Send Trap"
4. Input the IP address of the Nagios Server, everything else is pre-populated.