No Traps displayed in Web GUI | Waiting for Trap…

[Syndeticom]

What I am ultimately trying to achieve is to listen to several 3rd party, non-commercially used network devices; These all generate SNMP traps and forward them to my Nagios machine (192.168.1.2).

My Nagios machine receives all these traps, and then as soon as it receives them it forwards them to a monitoring software higher in the network topology (192.168.1.83).

The host (192.168.1.1) OS is CentOS 7 with Oracle VirtualBox as VM Manager. I am running Nagios XI (192.168.1.2) as a virtual machine, I downloaded the .ova file from here: https://www.nagios.com/downloads/nagios ... sxi-ova-64

Then I enabled the SNMP Trap sender by following this guide: https://assets.nagios.com/downloads/nag ... ios_XI.pdf

As the above-mentioned guide instructs:

You may need to configure snmptt on the Nagios XI server to use the MIBs your remote devices are using. This can be done via Admin > System Extensions > Manage MIBs. Click the Browse button to find the MIB to be added. Check the box Process trap and then click the Upload MIB button.

So, I downloaded these two MIBs: NAGIOS-NOTIFY-MIB.txt & NAGIOS-ROOT-MIB.txt from inside the Nagios Web GUI here: (Admin > Manage Componenets > SNMP Trap Sender > Edit Settings

I then uploaded (the Process Trap checkbox is checked) both MIBs through the Admin > Manage MIBs Web GUI.

I send a made-up trap (See below) from another host on the local network. The trap gets logged in the /var/log/snmptt/snmptt.log but:

1.) It takes about 2minutes for the event to display inside the .log file.
2.) It does not show in Web GUI; the message there still says “Waiting for Trap…”
3.) The forum thread related to similar issue does not seem to offer any conclusive solution: https://support.nagios.com/forum/viewto ... =16&t=9034


This is what the log file shows:

Code: Select all

Thu Mar  9 14:48:38 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.1.99 - Link up on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3
Thu Mar  9 14:49:18 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.1.99 - Link up on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3
Thu Mar  9 16:04:45 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:11:28 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:35:18 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 01.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:57:01 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" localhost - Link up on interface eth0.  Admin state: up.  Operational state: up
Thu Mar  9 16:59:10 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 02.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:59:55 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 03.  Admin state: $2.  Operational state: $3

My question is:

• 1.) Why isn’t anything showing in Web GUI;
  2.) Why is it taking so long for Nagios to log the Trap in /var/log/snmptt/snmptt.log?

And for the record, I am using TrapGen (http://trapgen.trapreceiver.com/) to generate files on my windows machine.

From Windows CMD I run the command:

Code: Select all

trapgen -f SNMP_test_2.txt

And the file “SNMP_test_2.txt” I use for the SNMP Trap contains:

Code: Select all

-d 192.168.1.2:162
-c public
-o .1.2.3.4.5.6.7.8.9.0
-i 192.168.1.99
-g 2
-s 23
-t 12445
-v 1.3.6.1.4.1.1824.1.0.0.1 STRING Trapgen test. Syndeticom 01

Where:

Code: Select all

-c <community string>
        -d <destinationIpOrHost[:portnumber]>
        -v <varbind OID> <varbind type> <varbind data>                 
        -o <sender's OID>
        -i <sender's IP address - V1 only>
        -g <generic type>
        -s <specific type>
        -t <timestamp>

I notice that he -o parameter is completely ignored by Nagios’ MIB and TrapTranslator, and it’s actually the -g that determines what kind of SNMP Trap it will be translated as. (This is not bothering me as much as learning WHAT/WHERE can I change this in Nagios so that I can translate my own SNMP traps )

[tgriep]

Can you post the following files from your XI server so we can view the SNMP settings?

Code: Select all

/etc/snmp/snmptt.conf
/etc/snmp/snmptt.ini
/usr/local/nagiosxi/var/corelog.newobjects

Also, can you run the following as root on the XI server and post the output as well?

Code: Select all

ls -l /usr/local/nagios/var/rw/
ps -ef |grep snmp

Thanks

[Syndeticom]

Thanks for the prompt response tgriep

Ok, here are the files, both snmptt conf & ini, but the corelog.newobjects is not there:

file_missing.PNG

Also, the requested outputs are as follows:

Code: Select all

ls -l /usr/local/nagios/var/rw/
total 0
prw-rw---- 1 nagios nagcmd 0 Mar 10 08:54 nagios.cmd
srw-rw---- 1 nagios nagcmd 0 Mar 10 08:54 nagios.qh

&

Code: Select all

ps -ef |grep snmp
root      1077     1  0 08:54 ?        00:00:00 /usr/sbin/snmptrapd -Lsd -p /var/run/snmptrapd.pid
root      1525     1  0 08:54 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
snmptt    1526  1525  0 08:54 ?        00:00:00 /usr/bin/perl /usr/sbin/snmptt --daemon
root      6374  1686  0 09:10 tty1     00:00:00 grep snmp

You are basically checking whether the relevant nagios process are running, and whether the configuration is set correctly, yes?

[tgriep]

If you look at the snmptt.conf file, the linkup and the linkdown traps you are using has the EXEC line commented out and also they are not using the correct command if they were uncommented to send data to the XI GUI.

The way it works, it the OID matches the settings, it runs the EXEC line and when that runs, it processes the TRAP so XI can process it.

Edit the /etc/snmp/snmptt.conf file and in the linkDown section, change this line from
#EXEC qpage -f TRAP notifygroup1 "Link down on interface $1. Admin state: $2. Operational state: $3"
to
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Link down on interface $1. Admin state: $2. Operational state: $3"


And in the LinkUp section, change this line from
#EXEC qpage -f TRAP notifygroup1 "Link up on interface $1. Admin state: $2. Operational state: $3"
to
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Link up on interface $1. Admin state: $2. Operational state: $3"


Save the file and restart the SNMPTT daemon by running
service snmptt restart


Then send a trap in and it should show up in the Admin > Unconfigured Objects menu in the XI GUI.
Let us know the outcome.

[Syndeticom]

A little progress - that is somewhat working . But not making much sense yet, I'll explain below.

Is there any way to make this SNMP Trap process faster? It takes tens of seconds between when the trap is generated on the client machine and the time when it's received/logged by Nagios. This reporting with such a delay will not sufficient if Nagios is to be given a green light in this project.

Now for the explanation I mentioned above:

This is the SNMP trap that is being sent by the client:

Code: Select all

inform
-d 192.168.1.2
-c private
-o 1.3.6.1.4.1.1824
-i 199.8.7.6
-g 2 #<<<<<<<<<<<<<<<<NOTE THIS LINE (This is a comment, it's not in the actual file)
-s 23
-t 12445
-v 1.3.6.1.4.1.1824.1.0.0.1 STRING "Syndeticom String"
-v 1.3.6.1.4.1.1824.1.0.0.1 COUNTER 11
-v 1.3.6.1.4.1.1824.1.0.0.1 GAUGE 22
-v 1.3.6.1.4.1.1824.1.0.0.1 INTEGER 33
-v 1.3.6.1.4.1.1824.1.0.0.1 ADDRESS 4.4.4.4
-v 1.3.6.1.4.1.1824.1.0.0.1 OID 1.2.3.4.5.6.7.8.9
-v 1.3.6.1.4.1.1824.1.0.0.1 TIMETICK 2233121

This is what the snmptt.log looks like now:

Code: Select all

Thu Mar  9 14:48:38 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.1.99 - Link up on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3
Thu Mar  9 14:49:18 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" 192.168.1.99 - Link up on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3
Thu Mar  9 16:04:45 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:11:28 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:35:18 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 01.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:57:01 2017 .1.3.6.1.6.3.1.1.5.4 Normal "Status Events" localhost - Link up on interface eth0.  Admin state: up.  Operational state: up
Thu Mar  9 16:59:10 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 02.  Admin state: $2.  Operational state: $3 
Thu Mar  9 16:59:55 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface Trapgen test. Syndeticom 03.  Admin state: $2.  Operational state: $3 
Thu Mar 16 15:16:28 2017 .1.3.6.1.6.3.1.1.5.2 Normal "Status Events" 192.168.1.99 - Device reinitialized (warmStart)
Thu Mar 16 15:36:00 2017 .1.3.6.1.6.3.1.1.5.2 Normal "Status Events" 192.168.1.99 - Device reinitialized (warmStart)
Thu Mar 16 15:39:21 2017 .1.3.6.1.6.3.1.1.5.2 Normal "Status Events" 192.168.1.99 - Device reinitialized (warmStart)
Thu Mar 16 15:41:11 2017 .1.3.6.1.6.3.1.1.5.2 Normal "Status Events" 192.168.1.99 - Device reinitialized (warmStart)
Thu Mar 16 15:50:56 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface \"Syndeticom String\".  Admin state: 83.  Operational state: 83 
Thu Mar 16 16:24:32 2017 .1.3.6.1.6.3.1.1.5.3 Normal "Status Events" 192.168.1.99 - Link down on interface \"Syndeticom String\".  Admin state: 11.  Operational state: 22 

And this is what the Web GUI SNMP Trap service shows in Firefox:

Code: Select all

Link down on interface "Syndeticom String". Admin state: 11. Operational state: 22 / enterprises.1824.1.0.0.1 ():"Syndeticom String" enterprises.1824.1.0.0.1 ():11 enterprises.1824.1.0.0.1 ():22 enterprises.1824.1.0.0.1 ():33 enterprises.1824.1.0.0.1 ():4

The thing that bothers me the most is that I do not understand how this trap is handled by Nagios; Followed closely by this:

The OID I am sending is

Code: Select all

-o 1.3.6.1.4.1.1824

right?
But SNMP only ever mentions

Code: Select all

.1.3.6.1.6.3.1.1.5.2

in the snmptt.log file - and on top of everything;
- note the .2 at the end above - that seems to correspond with "-g 2" line mentioned above. I feel like I am getting close to the core of the problem by realizing this relation.
Nagios web GUI displays yet another OID:

Code: Select all

enterprises.1824.1.0.0.1 ()

So basically it doesn't look like Nagios' snmptt.conf is really the place where Nagios looks and compares the OIDs - which are then acted upon.

I need to be able to effectively control what traps should be processed and displayed in GUI and what can be skipped and logged in the snmpttunknown.log - but I yet have to discover this handle. Is this making sense?
Most importantly, is this something I can look up in Nagios documents, because all the documents I've read so far never mention any configuration alterations; I am looking for the "what makes it go" if that's making sense.

Any additional info on the matter will be greatly appreciated. I also attached the latest snmptt.conf just to be sure there is enough background information to help me crack this nut.

[tgriep]

If your device is sending to this OID, 1.3.6.1.4.1.1824, the snmptt translator should mark it as unknown and put it in the /var/log/snmptt/snmpttunknown.log file.
If the OID is not in the snmptt.conf file, then is should put it in the unknown log.
The only option could be that in the bottom of the /etc/snmp/snmptt.ini file, you can specify multiple conf files and that OID is in another file.
Cnn you check the ini file?

The only other thought is your device is sending traps to both of those OID's.

If you look at this setting from the snmptt.conf file

Code: Select all

EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "Link down on interface $1. Admin state: $2. Operational state: $3"

and match it to the message in the GUI

Code: Select all

Link down on interface "Syndeticom String". Admin state: 11. Operational state: 22 / enterprises.1824.1.0.0.1 ():"Syndeticom String" enterprises.1824.1.0.0.1 ():11 enterprises.1824.1.0.0.1 ():22 enterprises.1824.1.0.0.1 ():33 enterprises.1824.1.0.0.1 ():4

The $1 from the EXEC line matches the first variable in your trap "-v 1.3.6.1.4.1.1824.1.0.0.1 STRING "Syndeticom String"
The $2 from the EXEC line matches the second variable in your trap "-v 1.3.6.1.4.1.1824.1.0.0.1 COUNTER 11"
The $3 from the EXEC line matches the third variable in your trap "-v 1.3.6.1.4.1.1824.1.0.0.1 GAUGE 22"
Those look OK but the rest of message has to be sent from the remote device so maybe it is not sending the trap in the correct format so can you provide more details on that?

It does take a few seconds to receive the trap and process it as it is a serial function and things run on a one second queue but I an guessing that the screen update is more or a refresh issue from that I feel.

Here are a few links to some KB artices that talk about the format of traps and how the server processes them.
https://support.nagios.com/kb/article.p ... php?id=558

[Syndeticom]

Thanks for all your input tgriep;

Is there any way to decrease the time it takes to process the trap in Nagios somewhere please?

[tgriep]

The Traps are put in a queue to be processed with all of the other alerts and the queue is processed as fast as it can so there isn't a way to speed it up.