snmptt is not translating trap in correct format

[[email protected]]

Hi

I have configure nagios core 4.0.7 with nrpe and its working perfectly fine and now I want extend nagios capability to monitor my SNMP device like switch router and ESXi host.

To that I have installed below RPM.

Code: Select all

[root@sov-srv-negios01 snmptt]# rpm -qa |grep snmp
net-snmp-utils-5.5-50.el6_6.1.x86_64
net-snmp-libs-5.5-50.el6_6.1.x86_64
net-snmp-devel-5.5-50.el6_6.1.x86_64
snmptt-1.4-0.9.beta2.el6.noarch
net-snmp-perl-5.5-50.el6_6.1.x86_64
net-snmp-5.5-50.el6_6.1.x86_64

I have configure SNMP TRAP daemon and my snmptrapd.conf file look like this

Code: Select all

[root@sov-srv-negios01 snmp]# more snmptrapd.conf

#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
logoption f /var/log/snmptrap.log
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler
#traphandle default /usr/sbin/snmptt
createUser -e 8000000001020304 arqadmin SHA mypassword AES
authuser log arqadmin

After this I have configured SNMPTT daemon and i have attached my snmp.ini file

I have convert VMware MIBS by below command

Code: Select all

# for i in `ls -lrt |awk '{print $9}'`; do snmpttconvertmib --in=$i --out=/etc/snmp/snmptt.conf.vmware --exec='/usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 2 '; done

I have also attched my snmptt.con.vmware file

I have installed NSTI package also and all this work but I am not able get snmptt logs in correct format.

When I am poweron/off VM in ESXi host I am getting below messages in my snmptt.log file

snmptt logs

Code: Select all

Mon Nov  3 14:16:25 2014 .1.3.6.1.4.1.6876.4.1.0.1 Normal "Status Events" hem-lab-esxi-10 - 1 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes
Mon Nov  3 14:17:14 2014 .1.3.6.1.4.1.6876.4.1.0.4 Normal "Status Events" hem-lab-esxi-10 - 4 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes
Mon Nov  3 14:17:44 2014 .1.3.6.1.4.1.6876.4.1.0.4 Normal "Status Events" hem-lab-esxi-10 - 4 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes
Mon Nov  3 14:18:48 2014 .1.3.6.1.4.1.6876.4.1.0.2 Normal "Status Events" hem-lab-esxi-10 - 2 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes

My snmptrap.log like this

Code: Select all

2014-11-03 14:16:25 hem-lab-esxi-10 [UDP: [10.128.27.40]:35774->[10.128.156.100]]:
.1.3.6.1.2.1.1.3.0 = Timeticks: (39882200) 4 days, 14:47:02.00  .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.4.1.6876.4.1.0.1 .1.3.6.1.4.1.6876.50.101.0 = INTEGER: 1 .1.3.6.1.4.1.6876.50.102.0 = STRING: "/vmfs/volumes/54468116-b03d8ea4-431d-
549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx"   .1.3.6.1.4.1.6876.2.1.1.2.20 = STRING: "test-centos-tempaltes"
2014-11-03 14:17:14 hem-lab-esxi-10 [UDP: [10.128.27.40]:35774->[10.128.156.100]]:
.1.3.6.1.2.1.1.3.0 = Timeticks: (39887200) 4 days, 14:47:52.00  .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.4.1.6876.4.1.0.4 .1.3.6.1.4.1.6876.50.101.0 = INTEGER: 4 .1.3.6.1.4.1.6876.50.102.0 = STRING: "/vmfs/volumes/54468116-b03d8ea4-431d-
549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx"   .1.3.6.1.4.1.6876.2.1.1.2.20 = STRING: "test-centos-tempaltes"
2014-11-03 14:17:44 hem-lab-esxi-10 [UDP: [10.128.27.40]:35774->[10.128.156.100]]:
.1.3.6.1.2.1.1.3.0 = Timeticks: (39890200) 4 days, 14:48:22.00  .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.4.1.6876.4.1.0.4 .1.3.6.1.4.1.6876.50.101.0 = INTEGER: 4 .1.3.6.1.4.1.6876.50.102.0 = STRING: "/vmfs/volumes/54468116-b03d8ea4-431d-
549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx"   .1.3.6.1.4.1.6876.2.1.1.2.20 = STRING: "test-centos-tempaltes"
2014-11-03 14:18:15 hem-lab-esxi-10 [UDP: [10.128.27.40]:35774->[10.128.156.100]]:
.1.3.6.1.2.1.1.3.0 = Timeticks: (39893200) 4 days, 14:48:52.00  .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.4.1.6876.4.90.0.401      .1.3.6.1.4.1.6876.4.30.3.0 = Hex-STRING: 07 DE 0B 03 0F 0E 26 00
2014-11-03 14:18:48 hem-lab-esxi-10 [UDP: [10.128.27.40]:35774->[10.128.156.100]]:
.1.3.6.1.2.1.1.3.0 = Timeticks: (39896500) 4 days, 14:49:25.00  .1.3.6.1.6.3.1.1.4.1.0 = OID: .1.3.6.1.4.1.6876.4.1.0.2 .1.3.6.1.4.1.6876.50.101.0 = INTEGER: 2 .1.3.6.1.4.1.6876.50.102.0 = STRING: "/vmfs/volumes/54468116-b03d8ea4-431d-
549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx"   .1.3.6.1.4.1.6876.2.1.1.2.20 = STRING: "test-centos-tempaltes"

When I run grep command to grep OID showing in snmptrap.log

Code: Select all

[root@sov-srv-negios01 snmp]# grep .1.3.6.1.4.1.6876.4.1.0.1 snmptt.conf.vmware
EVENT vmwVmPoweredOn .1.3.6.1.4.1.6876.4.1.0.1 "Status Events" Normal

[root@sov-srv-negios01 snmp]# grep .1.3.6.1.4.1.6876.4.1.0.2 snmptt.conf.vmware
EVENT vmwVmPoweredOff .1.3.6.1.4.1.6876.4.1.0.2 "Status Events" Normal

But Why this above line is not showing in snmptt.log

I want "EVENT vmwVmPoweredOff .1.3.6.1.4.1.6876.4.1.0.2 "Status Events" Normal" to shown snmptt.log.

Please tell me how can achive this

I have checked in my mysql database ist showing everything

My Database table information showing like this.

Code: Select all

EVENT vmwVmPoweredOff .1.3.6.1.4.1.6876.4.1.0.2 "Status Events" Normal

214 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:43:50.00 | Mon Nov  3 14:13:13 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 09 25 00                                   |        0 | 2014-11-03 14:13:16 |
| 215 | vmwVmPoweredOn    | .1.3.6.1.4.1.6876.4.1.0.1    | .1.3.6.1.4.1.6876.4.1.0.1    |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:47:02.00 | Mon Nov  3 14:16:25 2014 | 1 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes |        0 | 2014-11-03 14:16:26 |
| 216 | vmwVmHBDetected   | .1.3.6.1.4.1.6876.4.1.0.4    | .1.3.6.1.4.1.6876.4.1.0.4    |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:47:52.00 | Mon Nov  3 14:17:14 2014 | 4 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes |        0 | 2014-11-03 14:17:16 |
| 217 | vmwVmHBDetected   | .1.3.6.1.4.1.6876.4.1.0.4    | .1.3.6.1.4.1.6876.4.1.0.4    |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:48:22.00 | Mon Nov  3 14:17:44 2014 | 4 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes |        0 | 2014-11-03 14:17:46 |
| 218 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:48:52.00 | Mon Nov  3 14:18:15 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 0E 26 00                                   |        0 | 2014-11-03 14:18:16 |
| 219 | vmwVmPoweredOff   | .1.3.6.1.4.1.6876.4.1.0.2    | .1.3.6.1.4.1.6876.4.1.0.2    |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:49:25.00 | Mon Nov  3 14:18:48 2014 | 2 /vmfs/volumes/54468116-b03d8ea4-431d-549f3541e74a/test-centos-tempaltes/test-centos-tempaltes.vmx test-centos-tempaltes |        0 | 2014-11-03 14:18:52 |
| 220 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:53:54.00 | Mon Nov  3 14:23:17 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 13 28 00                                   |        0 | 2014-11-03 14:23:22 |
| 221 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:14:58:55.00 | Mon Nov  3 14:28:18 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 18 2A 00                                   |        0 | 2014-11-03 14:28:22 |
| 222 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:15:03:57.00 | Mon Nov  3 14:33:19 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 1D 2B 00                                   |        0 | 2014-11-03 14:33:22 |
| 223 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:15:08:59.00 | Mon Nov  3 14:38:21 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 22 2C 00                                   |        0 | 2014-11-03 14:38:22 |
| 224 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:15:14:01.00 | Mon Nov  3 14:43:23 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 27 2E 00                                   |        0 | 2014-11-03 14:43:27 |
| 225 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:15:19:02.00 | Mon Nov  3 14:48:24 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 2C 30 00                                   |        0 | 2014-11-03 14:48:27 |
| 226 | vmwCimOmHeartbeat | .1.3.6.1.4.1.6876.4.90.0.401 | .1.3.6.1.4.1.6876.4.90.0.401 |            |           | hem-lab-esxi-10 | 10.128.27.40 | Status Events | Normal   | 4:15:24:03.00 | Mon Nov  3 14:53:26 2014 | This notification, if the agent is so configured, will be sent  07 DE 0B 03 0F 31 31 00                                   |        0 | 2014-11-03 14:53:27 |
+-----+-------------------+------------------------------+------------------------------+------------+-----------+-----------------+--------------+---------------+----------+---------------+--------------------------+---------------------------------------------------------------------------------------------------------------------------+----------+---------------------+
226 rows in set (0.00 sec)

I want that my second column(vmwVmPoweredOff) to shown in snmptt.log

Please help me

Thanks

Sanjay

[rhassing]

Hi,

Maybe a few things that might help:

You could simplify the /etc/snmp/snmptrapd.conf file:

Code: Select all

authCommunity log,execute,net public
traphandle default /usr/sbin/snmptt

This means that snmptraps with a community 'public' are processed.

I noticed a vmwVmPoweredOn creates a critical snmptrap. I'm not sure if that is what you want, but that should be no problem.

My /etc/snmp/snmptt.ini file looks like this (without comments):

Code: Select all

[General]
snmptt_system_name = management
mode = standalone
multiple_event = 1
dns_enable = 1
strip_domain = 0
strip_domain_list = <<END
domain.com
END
resolve_value_ip_addresses = 0
net_snmp_perl_enable = 0
net_snmp_perl_best_guess = 0
translate_log_trap_oid = 0
translate_value_oids = 1
translate_enterprise_oid_format = 1
translate_trap_oid_format = 1
translate_varname_oid_format = 1
translate_integers = 1
mibs_environment = ALL
wildcard_expansion_separator = " "
allow_unsafe_regex = 0
remove_backslash_from_quotes = 0
dynamic_nodes = 0
description_mode = 0
description_clean = 1
threads_enable = 0
threads_max = 10

[DaemonMode]
daemon_fork = 1
daemon_uid = snmptt
pid_file = /var/run/snmptt.pid
spool_directory = /var/spool/snmptt/
sleep = 5
use_trap_time = 1
keep_unlogged_traps = 1
duplicate_trap_window = 0

[Logging]
stdout_enable = 0
log_enable = 1
log_file = /var/log/snmptt/snmptt.log
log_system_enable = 0
log_system_file = /var/log/snmptt/snmpttsystem.log
unknown_trap_log_enable = 1
unknown_trap_log_file = /var/log/snmptt/snmpttunknown.log
statistics_interval = 0
syslog_enable = 1
syslog_facility = local0
syslog_level_debug = <<END
END
syslog_level_info = <<END
END
syslog_level_notice = <<END
END
syslog_level_warning = <<END
END
syslog_level_err = <<END
END
syslog_level_crit = <<END
END
syslog_level_alert = <<END
END
syslog_level = warning
syslog_system_enable = 1
syslog_system_facility = local0
syslog_system_level = warning
[SQL]
db_translate_enterprise = 0
db_unknown_trap_format = '$-*'
sql_custom_columns = <<END
END
sql_custom_columns_unknown = <<END
END
mysql_dbi_enable = 0
mysql_dbi_host = localhost
mysql_dbi_port = 3306
mysql_dbi_database = snmptt
mysql_dbi_table = snmptt
mysql_dbi_table_unknown = snmptt_unknown
mysql_dbi_table_statistics = 
mysql_dbi_username = snmpttuser
mysql_dbi_password = password
mysql_ping_on_insert = 1
mysql_ping_interval = 300
postgresql_dbi_enable = 0
postgresql_dbi_module = 0
postgresql_dbi_hostport_enable = 0
postgresql_dbi_host = localhost
postgresql_dbi_port = 5432
postgresql_dbi_database = snmptt
postgresql_dbi_table_unknown = snmptt_unknown
postgresql_dbi_table_statistics = 
postgresql_dbi_table = snmptt
postgresql_dbi_username = snmpttuser
postgresql_dbi_password = password
postgresql_ping_on_insert = 1
postgresql_ping_interval = 300
dbd_odbc_enable = 0
dbd_odbc_dsn = snmptt
dbd_odbc_table = snmptt
dbd_odbc_table_unknown = snmptt_unknown
dbd_odbc_table_statistics = 
dbd_odbc_username = snmptt
dbd_odbc_password = password
dbd_odbc_ping_on_insert = 1
dbd_odbc_ping_interval = 300

[Exec]
exec_enable = 1
pre_exec_enable = 1
unknown_trap_exec = 
unknown_trap_exec_format = 
exec_escape = 1

[Debugging]
DEBUGGING = 1
DEBUGGING_FILE = /var/log/snmptt/snmptt.debug
DEBUGGING_FILE_HANDLER = /var/log/snmptt/snmptthandler.debug

[TrapFiles]
snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
/etc/snmp/snmptt.conf.foundry
/etc/snmp/snmptt.conf.cpqida
/etc/snmp/snmptt.conf.cpqnic
/etc/snmp/snmptt.conf.cpqrack
/etc/snmp/snmptt.conf.CPQRPM.MIB
/etc/snmp/snmptt.conf.gbe2
/etc/snmp/snmptt.conf.iscsi
/etc/snmp/snmptt.conf.J4903A
/etc/snmp/snmptt.conf.msa
/etc/snmp/snmptt.conf.netapp
/etc/snmp/snmptt.conf.netscreen
/etc/snmp/snmptt.conf.ups
/etc/snmp/snmptt.conf.vmware
/etc/snmp/snmptt.conf.cmmc
END

Make sure you add your own snmptt.conf file(s)

The snmptrapd should be started with the following options (so it does not try to translate itself):

Code: Select all

OPTIONS="-Lsd -p /var/run/snmptrapd.pid -m ALL -On"

Now you should have some different log files. First of all you should be able to your snmptt log files in /var/log/snmptt/ and it should also give you the default snmptrap in your syslog file.

Just for my curiosity, what distribution are you using?

Best regards,
Rob Hassing

[sreinhardt]

Thanks for the input Rob, some good points there! It appears that you presently have snmptrapd logging to an snmptt.log file and snmptt logs known traps to a file named the same, but in a different location. It is very confusing to try and understand which file you are actually referencing. However one thing I would note is that snmptt's snmptt.log MIGHT have the exec information in it for you, and a debug file definitely would. Anything from snmptrapd most definitely would not have the information you are looking for as it never handles exec lines of any sort. Could you run us through what you feel should be happening one more time with the snmptrapd file that you have named snmptt.log renamed for explanation purposes to snmptrapd.log, or append the full path so we can't get confused?

[[email protected]]

Hi Rob

Thanks for reply and Now I am able to get right messages in snmptt.log file my messages are like this

Tue Nov 4 10:59:35 2014 .1.3.6.1.4.1.6876.4.1.0.1 Normal "Status Events" hem-lab-esxi-10 - VmPoweredOn VM test-centos-tempaltes
Tue Nov 4 11:00:26 2014 .1.3.6.1.4.1.6876.4.1.0.4 Normal "Status Events" hem-lab-esxi-10 - VM test-centos-tempaltes detects or regains the required number of guest heartbeats
Tue Nov 4 11:00:56 2014 .1.3.6.1.4.1.6876.4.1.0.4 Normal "Status Events" hem-lab-esxi-10 - VM test-centos-tempaltes detects or regains the required number of guest heartbeats
Tue Nov 4 11:01:09 2014 .1.3.6.1.4.1.6876.4.1.0.2 Normal "Status Events" hem-lab-esxi-10 - VmPoweredOff test-centos-tempaltes
Tue Nov 4 11:03:36 2014 .1.3.6.1.4.1.6876.4.90.0.401 Normal "Status Events" hem-lab-esxi-10 - This notification, for vmwCimOmHeartbeat live

I have only edited my snmptt.conf.vmware file

File looking like this.

EVENT vmwVmPoweredOn .1.3.6.1.4.1.6876.4.1.0.1 "Status Events" Normal
FORMAT VM $3 VmPoweredOn
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 0 "$*"
SDESC

This trap is sent when a virtual machine is powered on from a suspended
or a powered off state. The origin of this event can be several:
for instance may be operator initiated, existing vmx process reconnects to control subsystem.
NOTE: vms powered up due to VMotion are not reported. Upon receiving this notification client applications should
poll the vmwVmTable to obtain current status.
Variables:
1: vmwVmID
2: vmwVmConfigFilePath
3: vmwVmDisplayName
EDESC
#
#
#
EVENT vmwVmPoweredOff .1.3.6.1.4.1.6876.4.1.0.2 "Status Events" Normal
FORMAT VM $3 VmPoweredOff
EXEC /usr/local/nagios/libexec/eventhandlers/submit_check_result $r TRAP 2 "$*"
SDESC

This trap is sent when a virtual machine is powered off. The origin of this event can be several:
for instance may be operator initiated, vmx process terminating abnormally. NOTE: vms powered down due
to VMotion are not reported. Upon receiving this notification client applications should
poll the vmwVmTable to obtain current status.

[rhassing]

Hello Sanjay,

Good to hear things are working for you now.

Have fun with your snmptrap environment

Best regards,
Rob

[lgroschen]

I had to bribe Rob with some beer to make him tell me all his SNMP secrets! He knows his stuff

[eloyd]

Bribe Rob with beer? Isn't that like saying you had to bribe a fish with water?

[abrist]

Bribe Rob with beer? Isn't that like saying you had to bribe a fish with water?

If that was case, wouldn't alcoholics be a protected class?
PS: I am closing this before I get myself in trouble . . . .



[edit: hahah -Luke]