Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Fri Apr 5 17:07:54 2013 .1.3.6.1.4.1.2540.1.1.1.23.0.45.2 Normal "Status Events" 192.168.0.1 - A device has gone into or come out of maintenance mode 0
The nagios event log shows two contemporary events.
I came across this issue, gosh it must be two years ago now. Nagios isn't the problem here, the same trap is actually being submitted twice to Nagios.
From hazy memory my particular problem was SNMPTT being configured to run in daemon mode but being executed in standalone mode, but this problem can also be caused by having multiple snmptrapd handlers submitting to nagios. A good way to test this would be to run the python trap submission script manually and observe that Nagios processes only one "trap".
Because I can't remember exactly what the root cause was I can't provide precise details on what I did to solve it, I do remember it was a difficult one to track down.
Once again jsmurphy with a solid post! I would completely agree that this is likely either the device sending traps twice, or more likely snmptt deciding it needs to duplicate the event to nagios twice. Using the smnptrap sending script is an excellent way to test and verify if it snmptt is duplicating the events, as you should also likely see the same duplication in nagios logs. If you do not see duplicate entries again, the first step should be to verify that your device is not for some reason sending events twice.
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
I verified my logs and I confirm the informations reported in my first post.
Regarding one single alarm:
snmtrapd log reports just one message
snmptt log reports just one message
nagios log reports two identical messages, at the same time.
The strange thing is that Nagios monitors properly other traps sent by the same device (but with different OID).
I think a wrong snmtt configuration due always a double trap.
I report the snmptt config file
EVENT dbtMaintModeTrap .1.3.6.1.4.1.2540.1.1.1.23.0.45.2 "Status Events" Normal
FORMAT A device has gone into or come out of maintenance mode $*
PREEXEC /bin/echo $1 | /bin/sed -e s/0/"ON"/ -e s/1/"OFF"/ # --> $p1
EXEC /usr/lib64/nagios/plugins/eventhandlers/submit_check_result $r TRAP 1 "MaintMode $p1 $3"
Did you try running the script manually from the cli as suggested by jsmurphy?
jsmurphy wrote: A good way to test this would be to run the python trap submission script manually and observe that Nagios processes only one "trap".
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
mrninni wrote:
I verified my logs and I confirm the informations reported in my first post.
Regarding one single alarm:
snmtrapd log reports just one message
snmptt log reports just one message
nagios log reports two identical messages, at the same time.
I just remembered what my problem was! You my friend are in luck as on a regular day I can't remember what I ate for breakfast. Today I ate errr... I don't know... but as for your problem:
You have two different mibs that have the same OID in them, by default SNMPTT will generate a message for each OID entry, but it will only log one entry in it's internal logs (because SNMPTT logs INPUT not OUTPUT). You can solve this in two ways, either remove one of the OID entries from the SNMPTT translation file or there is a configuration option for SNMPTT to tell it to stop after finding the first OID match.
From the SNMPTT help document:
...There can be multiple entries for the same trap OID in the configuration file. If multiple_event is enabled in the snmptt.ini, then it will process all matching traps. If multiple_event is disabled, only the first matching entry will be used.
By running manually the script (/usr/lib64/nagios/plugins/eventhandlers/submit_check_result 192.168.0.1 TRAP 1 "MaintMode test") the Nagios behaviour is ok, just one log is reported
The SNMPTT conf file of this device contains only 2 different OID:
The last solutions seems to set the SNMPTT.INI file as you reported: multiple_events=0.
I have to check another detail... Nagios runs in a clustered system (2 servers) with a single virtual IP address (trap destination set in the device). Maybe the device is set with both IP adresses of the two servers and not the virtual one.
@jsmurphy - Have you been knighted yet? As always, thanks for the help.
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
mrninni wrote:Hi all,
The SNMPTT conf file of this device contains only 2 different OID:
It's worth noting SNMPTT doesn't care what device it's from, if you have the same oid in ANY device mibs it will process all of them for the same oid. So if you have, as an example, Cisco switches and juniper firewalls and import the basic SNMPv2 mib for both (sometimes the same mib has different names from different vendors), then you will have ingested the same set of oids twice.
So whenever a trap is generated for an OID from that set of mibs it will process the translations from for every match regardless of device. This may not be your problem but it's worth keeping in mind
@jsmurphy - Have you been knighted yet? As always, thanks for the help.
I verified.. the device sent 2 traps to both clustered server.
Problem solved. Thanks.
So if you have, as an example, Cisco switches and juniper firewalls and import the basic SNMPv2 mib for both (sometimes the same mib has different names from different vendors), then you will have ingested the same set of oids twice.
Thankyou for the explanation.
I was (wrongly) sure about the different "enterprise" OID... for example all Cisco traps should begin with 1.3.6.1.4.1.9
and Juniper traps with 1.3.6.1.4.1.2636
As always, thanks for the help.
