Nagios Support Forum

Posted: **Thu Oct 31, 2013 10:23 am**

Having just one in there could mean that the files are getting reaped now, or that nothing is allowed in, unfortunately its tough to tell at this point. Slansing actually meant just above that event log portion is the syslog and flatfile logging configurations. Make sure that both known and unknown traps are set to be logged. You can also set debug options in trapd and snmptt and get more information, this would be a pretty good idea to see if traps are coming in, put in spool, and how they are being handled. You should end up with /var/log/snmptt/*.debug files.

Posted: **Thu Oct 31, 2013 10:32 am**

Hmm, that doc tells me to edit the snmptt.ini file variables eventlog_enable, eventlog_type, and eventlog_system_enable. Those don't exist in my .ini, but i did find some entries for logging, which are enabled already.

Code: Select all

# Set to 1 to enable text logging of *TRAPS*.  Make sure you specify a log_file
# location
log_enable = 1

# Log file location.  The COMPLETE path and filename.  Ex: '/var/log/snmptt/snmptt.log'
log_file = /var/log/snmptt/snmptt.log

# Set to 1 to enable text logging of *SNMPTT system errors*.  Make sure you
# specify a log_system_file location
log_system_enable = 1

# Log file location.  The COMPLETE path and filename.
# Ex: '/var/log/snmptt/snmpttsystem.log'
log_system_file = /var/log/snmptt/snmpttsystem.log

# Set to 1 to enable logging of unknown traps.  This should normally be left off
# as the file could grow large quickly.  Used primarily for troubleshooting.  If
# you have defined a trap in snmptt.conf, but it is not executing, enable this to
# see if it is being considered an unknown trap due to an incorrect entry or
# simply missing from the snmptt.conf file.
# Unknown traps can be logged either a text file, a SQL table or both.
# See SQL section to define a SQL table to log unknown traps to.
unknown_trap_log_enable = 1

# Unknown trap log file location.  The COMPLETE path and filename.
# Ex: '/var/log/snmptt/snmpttunknown.log'
# Leave blank to disable logging to text file if logging to SQL is enabled
# for unknown traps
unknown_trap_log_file = /var/log/snmptt/snmpttunknown.log

# How often in seconds statistics should be logged to syslog or the event log.
# Set to 0 to disable
# 1 hour = 216000
# 12 hours = 2592000
# 24 hours = 5184000
statistics_interval = 0

Posted: **Thu Oct 31, 2013 10:54 am**

So considering you have both known and unknown traps being logged, those files not being created tells me that you are likely not actually receiving traps. Considering its a udp protocol it is not unheard of for tests to send and get blocked by iptables without any knowledge of the sender.

Towards the end of snmptt.ini let's enable debugging and see what comes up.

DEBUGGING = 2 (currently set to 0)
Comment the debug_file line with no file and uncomment the one with a file specified.
Do the same for the debug_file_handler lines.
restart snmptt

Send a few test traps, and take a look at the files in /var/log/snmptt/*.debug. Attaching them here or posting a chunk of output would be helpful too!

Posted: **Thu Oct 31, 2013 11:21 am**

Here you go. This debug file looked the same before and after the test traps were sent.

Code: Select all

[root@lisl-ngos-01-pv snmptt]# cat snmptt.debug
********** SNMPTT v1.3 started: Thu Oct 31 11:00:25 2013 **********

********** Net-SNMP version 5.05 Perl module enabled **********

********** DNS enabled **********


        Loading /etc/snmp/snmptt.conf
        Finished loading 1333 lines from /etc/snmp/snmptt.conf

Finished loading configuration files

Processing memory copy of configuration files
62 EVENTs found
62 EVENTs found that contain at least the mandatory FORMAT line
Finished processing memory copy of configuration files

==========================================================
Printing out all the events in hash table:

Event: .1.3.6.1.2.1.16.0.1 => risingAlarm,Status Events,Normal,The SNMP trap that is generated when an alarm $*,
Event: .1.3.6.1.2.1.16.0.2 => fallingAlarm,Status Events,Normal,The SNMP trap that is generated when an alarm $*,
Event: .1.3.6.1.2.1.17.0.1 => newRoot,Status Events,Normal,The newRoot trap indicates that the sending agent has $*,
Event: .1.3.6.1.2.1.17.0.2 => topologyChange,Status Events,Normal,A topologyChange trap is sent by a bridge when any of $*,
Event: .1.3.6.1.2.1.55.2.0.1 => ipv6IfStateChange,Status Events,Normal,An ipv6IfStateChange notification signifies $*,
Event: .1.3.6.1.2.1.63.2.0.1 => schedActionFailure,Status Events,Normal,This notification is generated whenever the invocation of a $*,
Event: .1.3.6.1.2.1.64.2.0.1 => smScriptAbort,Status Events,Normal,This notification is generated whenever a running script $*,
Event: .1.3.6.1.2.1.64.2.0.2 => smScriptResult,Status Events,Normal,This notification can be used by scripts to notify other $*,
Event: .1.3.6.1.2.1.64.2.0.3 => smScriptException,Status Events,Normal,This notification can be used by scripts to notify other $*,
Event: .1.3.6.1.2.1.88.2.0.1 => mteTriggerFired,Status Events,Normal,Notification that the trigger indicated by the object $*,
Event: .1.3.6.1.2.1.88.2.0.2 => mteTriggerRising,Status Events,Normal,Notification that the rising threshold was met for triggers $*,
Event: .1.3.6.1.2.1.88.2.0.3 => mteTriggerFalling,Status Events,Normal,Notification that the falling threshold was met for triggers $*,
Event: .1.3.6.1.2.1.88.2.0.4 => mteTriggerFailure,Status Events,Normal,Notification that an attempt to check a trigger has failed. $*,
Event: .1.3.6.1.2.1.88.2.0.5 => mteEventSetFailure,Status Events,Normal,Notification that an attempt to do a set in response to an $*,
Event: .1.3.6.1.4.1.20006.1.5 => nHostEvent,Status Events,Normal,The SNMP trap that is generated as a result of an event with the host $*,
Event: .1.3.6.1.4.1.20006.1.6 => nHostNotify,Status Events,Normal,The SNMP trap that is generated as a result of an event requiring $*,
Event: .1.3.6.1.4.1.20006.1.7 => nSvcEvent,Status Events,Normal,The SNMP trap that is generated as a result of an event with the service $*,
Event: .1.3.6.1.4.1.20006.1.8 => nSvcNotify,Status Events,Normal,The SNMP trap that is generated as a result of an event requiring $*,
Event: .1.3.6.1.4.1.2021.251.1 => ucdStart,Status Events,Normal,This trap could in principle be sent when the agent start $*,
Event: .1.3.6.1.4.1.2021.251.2 => ucdShutdown,Status Events,Normal,This trap is sent when the agent terminates $*,
Event: .1.3.6.1.4.1.20632.5.1.10 => ServerDown,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.11 => OpenConnsLimitReached,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.12 => CpsLimitReached,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.13 => SslTPSLimitReached,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.14 => HighAvailabilityStatus,Status Events,Normal,Severity: Information  $*,
Event: .1.3.6.1.4.1.20632.5.1.15 => CpuTempCritical,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.16 => MinServerThresholdReached,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.2 => CpuFanDead,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.3 => SysFanDead,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.4 => CpuTempHigh,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.5 => FirmwareStorageHigh,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.6 => LogsStorageHigh,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.7 => ServiceUp,Status Events,Normal,Severity: Information $*,
Event: .1.3.6.1.4.1.20632.5.1.8 => ServiceDown,Status Events,Normal,Severity: Alert $*,
Event: .1.3.6.1.4.1.20632.5.1.9 => ServerUp,Status Events,Normal,Severity: Information $*,
Event: .1.3.6.1.4.1.3417.2.1.2.0.1 => deviceSensorTrap,Status Events,Normal,The sensor value warrants a notification. $*,
Event: .1.3.6.1.4.1.3417.2.1.2.0.1 => deviceSensorTrap,Status Events,Normal,The sensor value warrants a notification. $*,
Event: .1.3.6.1.4.1.3417.2.12.2.0.1 => bluecoatSgHealthMonitorTrap,Status Events,Normal,A notification is generated when the health $*,
Event: .1.3.6.1.4.1.3417.2.12.2.0.1 => bluecoatSgHealthMonitorTrap,Status Events,Normal,A notification is generated when the health $*,
Event: .1.3.6.1.4.1.3417.2.13.2.0.1 => bluecoatSgFailoverTrap,Status Events,Normal,A notification is generated when the failover state $*,
Event: .1.3.6.1.4.1.3417.2.13.2.0.1 => bluecoatSgFailoverTrap,Status Events,Normal,A notification is generated when the failover state $*,
Event: .1.3.6.1.4.1.3417.2.14.2.0.1 => sgICAPTrap,Status Events,Normal,A notification that represents an ICAP-  $*,
Event: .1.3.6.1.4.1.3417.2.14.2.0.1 => sgICAPTrap,Status Events,Normal,A notification that represents an ICAP-  $*,
Event: .1.3.6.1.4.1.3417.2.2.2.0.1 => deviceDiskTrap,Status Events,Normal,The disk status warrants a notification. $*,
Event: .1.3.6.1.4.1.3417.2.2.2.0.1 => deviceDiskTrap,Status Events,Normal,The disk status warrants a notification. $*,
Event: .1.3.6.1.4.1.3417.2.3.2.0.1 => deviceAttackTrap,Status Events,Normal,At the start of an attack a notification is $*,
Event: .1.3.6.1.4.1.3417.2.3.2.0.1 => deviceAttackTrap,Status Events,Normal,At the start of an attack a notification is $*,
Event: .1.3.6.1.4.1.3417.2.4.2.0.1 => deviceUsageTrap,Status Events,Normal,A notification is sent when the deviceUsagePercent $*,
Event: .1.3.6.1.4.1.3417.2.4.2.0.1 => deviceUsageTrap,Status Events,Normal,A notification is sent when the deviceUsagePercent $*,
Event: .1.3.6.1.4.1.3417.2.6.2.0.1 => devicePolicyTrap,Status Events,Normal,A notification is generated when triggered by policy $*,
Event: .1.3.6.1.4.1.3417.2.6.2.0.1 => devicePolicyTrap,Status Events,Normal,A notification is generated when triggered by policy $*,
Event: .1.3.6.1.4.1.3417.2.7.2.0.1 => deviceHealthCheckTrap,Status Events,Normal,A notification is generated when the health $*,
Event: .1.3.6.1.4.1.3417.2.7.2.0.1 => deviceHealthCheckTrap,Status Events,Normal,A notification is generated when the health $*,
Event: .1.3.6.1.4.1.8072.2.3.0.1 => netSnmpExampleHeartbeatNotification,Status Events,Normal,An example notification, used to illustrate the $*,
Event: .1.3.6.1.4.1.8072.4.0.1 => nsNotifyStart,Status Events,Normal,An indication that the agent has started running. $*,
Event: .1.3.6.1.4.1.8072.4.0.2 => nsNotifyShutdown,Status Events,Normal,An indication that the agent is in the process of being shut down. $*,
Event: .1.3.6.1.4.1.8072.4.0.3 => nsNotifyRestart,Status Events,Normal,An indication that the agent has been restarted. $*,
Event: .1.3.6.1.6.3.1.1.5.1 => coldStart,Status Events,Normal,A coldStart trap signifies that the SNMP entity, $*,
Event: .1.3.6.1.6.3.1.1.5.2 => warmStart,Status Events,Normal,A warmStart trap signifies that the SNMP entity, $*,
Event: .1.3.6.1.6.3.1.1.5.3 => linkDown,Status Events,Normal,A linkDown trap signifies that the SNMP entity, acting in $*,
Event: .1.3.6.1.6.3.1.1.5.4 => linkUp,Status Events,Normal,A linkUp trap signifies that the SNMP entity, acting in an $*,
Event: .1.3.6.1.6.3.1.1.5.5 => authenticationFailure,Status Events,Normal,An authenticationFailure trap signifies that the SNMP $*,

Finished printing out all events in hash table
==========================================================
cwd: /
Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

Sleeping for 5 seconds

You have new mail in /var/spool/mail/root
[root@lisl-ngos-01-pv snmptt]# ls
snmptt.debug  snmpttsystem.log
[root@lisl-ngos-01-pv snmptt]#

Posted: **Fri Nov 01, 2013 9:15 am**

Well it clearly seems to be reaping some files, and then sleeping for quite some time. Could you tar the entire /etc/snmp directory and pm it to me please? I'd like to have a look at it as a whole and see if there is anything that needs changing.

Posted: **Fri Nov 01, 2013 9:59 am**

Unless you set it back already, it seems that debugging was not enabled in snmptt.ini. Otherwise everything certainly appears in order. After making the changes, give snmptt service a reboot.

Code: Select all

[Debugging]
# 0 - do not output messages
# 1 - output some basic messages
# 2 - out all messages
DEBUGGING = 0

# Debugging file - SNMPTT
# Location of debugging output file.  Leave blank to default to STDOUT (good for
# standalone mode, or daemon mode without forking)
DEBUGGING_FILE = 
#DEBUGGING_FILE = /var/log/snmptt/snmptt.debug

# Debugging file - SNMPTTHANDLER
# Location of debugging output file.  Leave blank to default to STDOUT
DEBUGGING_FILE_HANDLER = 
#DEBUGGING_FILE_HANDLER = /var/log/snmptt/snmptthandler.debug

Should read:

Code: Select all

[Debugging]
# 0 - do not output messages
# 1 - output some basic messages
# 2 - out all messages
DEBUGGING = 2

# Debugging file - SNMPTT
# Location of debugging output file.  Leave blank to default to STDOUT (good for
# standalone mode, or daemon mode without forking)
#DEBUGGING_FILE = 
DEBUGGING_FILE = /var/log/snmptt/snmptt.debug

# Debugging file - SNMPTTHANDLER
# Location of debugging output file.  Leave blank to default to STDOUT
#DEBUGGING_FILE_HANDLER = 
DEBUGGING_FILE_HANDLER = /var/log/snmptt/snmptthandler.debug

Posted: **Fri Nov 01, 2013 10:16 am**

Ah, I shut it off yesterday before I left for the day, since all I was getting was sleeping for 5 seconds over and over and the debug file was getting rather long with just that repeated over and over. Debugging was enabled when we sent the test traps yesterday, though.

Posted: **Fri Nov 01, 2013 12:19 pm**

Do you need me to turn debugging back on and try the test traps again? I haven't made any changes (other than turning debugging back off again) since yesterday, so I can't imagine the result would be any different.

Posted: **Fri Nov 01, 2013 12:36 pm**

Nope, I have just been mulling around on this one. We had an issue today, you don't happen to be trying to sent snmpv3 traps do you? They are still v2 in your environment correct?

Posted: **Tue Nov 05, 2013 10:43 am**

Sorry, was out yesterday. Yeah, they are snmpv2 traps.

Nagios Support Forum

Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps

Re: Monitoring Traps