Home » Categories » Multiple Categories

SNMP Trap - snmptt Service

snmptt Service

The snmptt service is what processes traps that have been placed into the spool directory by the snmptrapd service. "Processing Traps" means to look at the spooled trap and determine if it needs to be submitted to Nagios and how.

Before proceeding, you would have followed the troubleshooting article "snmptrapd Service" as this previous article ensures that the traps are being spooled.

 

 

SNMP Trap Troubleshooting Notes

If you are up to this stage of troubleshooting SNMP Traps, then it is clear that your server is correctly receiving SNMP Traps from remote devices.

The majority of problems from this point forward have to do with how to configure snmptt so it correctly identify traps and submits them to Nagios.

Trying to explain how to configure snmptt is not something that can be quickly explained in a troubleshooting article. In particular it can be hard to start troubleshooting your problems with a device (like a UPS or Printer) as there are so many ways SNMP traps can be configured.

Instead, a detailed tutorial on SNMP Traps has been created. The purpose of the tutorial is to walk you through a test environment and explain how each part of the configuration works. Specifically it focuses on a easy to setup test environment which allows you to follow the tutorial step by step.

Once you've completed the tutorial, you will have a solid understanding of how SNMP Traps work with Nagios and you will then be able to more easily troubleshoot the problem you are having with your device.

The tutorial is located here under General Topics > SNMP > SNMP Trap Tutorial.

 

The remainder of this article will be to identify specific items to look at to understand what could be causing problems with the snmptt service and it's integration with Nagios.

 

 

Is The snmptt Service Running?

Execute the command for your operating system (OS):

 

RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20

systemctl status snmptt.service

 

If snmptt is running, it should produce output like:

snmptt (pid  11612) is running...

 

If snmptt is NOT running, it will produce output like:

snmptt is stopped

 

If the snmptt service is NOT running, this means that spooled SNMP Traps will not be processed and the spool folder will slowly fill up. To ENABLE snmptt on boot and to start it, execute the following commands depending on your OS:

 

RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20

systemctl enable snmptt.service
systemctl start snmptt.service

 

 

If snmptt is NOT installed, it will produce output like:

snmptt: unrecognized service

 

If you find that snmptt is not installed, you need to follow the guide How to Integrate SNMP Traps With Nagios XI.

 

 

snmptt Log Files

There are three log files that snmptt can generate:

  • /var/log/snmptt/snmpttsystem.log
    • This records what happens when the service starts. Information like version, config files used, UID.
  • /var/log/snmptt/snmptt.log
    • Traps that have been received AND detected as a trap to perform an action with.
  • /var/log/snmptt/snmpttunknown.log
    • Traps that have been received AND ignored.

 

 

snmptt Configuration Files

All of the settings relevant to how snmptt works are stored in the file:

/etc/snmp/snmptt.ini

 

The most important settings are at the end of the file in the [TrapFiles] section:

[TrapFiles]
# A list of snmptt.conf files (this is NOT the snmptrapd.conf file). The COMPLETE path
# and filename. Ex: '/etc/snmp/snmptt.conf'
snmptt_conf_files = <<END
/etc/snmp/snmptt.conf
END

 

Any files defined in this list are what snmptt looks at to determine if it needs to perform an action.

On a fresh install, only the snmptt.conf file is listed.

/etc/snmp/snmptt.conf

 

With Nagios XI, MIB files can be uploaded via Admin > System Extensions > Manage MIBs.

When a MIB is uploaded and the "Add to SNMPTT radio" is selected, traps identified in the MIB will be added to the /etc/snmp/snmptt.conf file.

 

NOTE: After uploading a MIB file the snmptt service is restarted to use the new traps it processed. Refer to the restart snmptt service steps you need to manually restart it.

 

 

Restart snmptt Service

Execute the command for your operating system (OS):

 

RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20

systemctl restart snmptt.service

 

 

Trap Files

Trap Files contain a list of SNMP Traps that snmptt is allowed to process. Basically this means that if the OID or MIB object is found in the Trap file, do something with it. "Do something" in relation to Nagios means to submit the result to a passive check in Nagios.

Each OID or MIB object referenced in a trap file usually corresponds with three lines, EVENT, FORMAT and EXEC. Additionally description text can also be contained between SDESC and EDESC. Here is an example:

 

EVENT nSvcEvent .1.3.6.1.4.1.20006.1.7 "Status Events" Normal
FORMAT The SNMP trap that is generated as a result of an event with the service $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated as a result of an event with the service $*"
SDESC
The SNMP trap that is generated as a result of an event with the service
in Nagios.
Variables:
1: nHostname
2: nHostStateID
3: nSvcDesc
4: nSvcStateID
5: nSvcAttempt
6: nSvcDurationSec
7: nSvcGroupName
8: nSvcLastCheck
9: nSvcLastChange
10: nSvcOutput
EDESC - See more at: http://support.nagios.com/kb/article.php?id=77&show_category=14#sthash.QeiAnSRJ.dpuf
EVENT nSvcEvent .1.3.6.1.4.1.20006.1.7 "Status Events" Normal
FORMAT The SNMP trap that is generated as a result of an event with the service $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated as a result of an event with the service $*"
SDESC
The SNMP trap that is generated as a result of an event with the service
in Nagios.
Variables:
1: nHostname
2: nHostStateID
3: nSvcDesc
4: nSvcStateID
5: nSvcAttempt
6: nSvcDurationSec
7: nSvcGroupName
8: nSvcLastCheck
9: nSvcLastChange
10: nSvcOutput
EDESC

 

Please refer to the SNMP Trap Tutorial for a detailed explanation.

NOTE: If you make any changes to the Trap files, the snmptt service needs to be restarted to reload the changes.

 

 

Processing Traps

When a snmptt matches a spooled trapped against an OID or MIB object in a trap file:

  • The FORMAT line will log this in the file:
    • /var/log/snmptt/snmptt.log
  • The EXEC line will use the snmptraphandling.py script to submit the trap to Nagios using the command pipe (nagios.cmd) as a passive check result.

 

Nagios will do one of two things when it receives the passive check result:

IF there is already an “SNMP Traps” service for the host being targeted:

It will update that service and it will NOT log anything in nagios.log (this can be enabled)

IF there is NOT an “SNMP Traps” service for the host being targeted:

The following entries will be logged in nagios.log

[1416634449] Warning: Passive check result was received for service 'SNMP Traps' on host 'snmpsender', but the service could not be found!

[1416634449] Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;snmpsender;SNMP Traps;0;The SNMP trap that is generated as a result of an event with the service CentOS Users 1 USERS WARNING - 1 users currently logged in / nSvcHostname (OCTETSTR):CentOS nSvcDesc (OCTETSTR):Users nSvcStateID (INTEGER):0 nSvcOutput (OCTETSTR):USERS OK - 0 users currently logged in

[1416634449] External command error: Command failed

The "SNMP Traps" service will appear as an Unconfigured object under Admin > Monitoring Config > Unconfigured Objects

- See more at: http://support.nagios.com/kb/article.php?id=77&show_category=14#sthash.QeiAnSRJ.dpuf

IF there is already an "SNMP Traps" service for the host being targeted:

It will update that service and it will NOT log anything in nagios.log (this can be enabled)

IF there is NOT an "SNMP Traps" service for the host being targeted:

The following entries will be logged in nagios.log

[1416634449] Warning:  Passive check result was received for service 'SNMP Traps' on host 'snmpsender', but the service could not be found!
[1416634449] Error: External command failed -> PROCESS_SERVICE_CHECK_RESULT;snmpsender;SNMP Traps;0;The SNMP trap that is generated as a result of an event with the service CentOS Users 1 USERS WARNING - 1 users currently logged in / nSvcHostname (OCTETSTR):CentOS nSvcDesc (OCTETSTR):Users nSvcStateID (INTEGER):0 nSvcOutput (OCTETSTR):USERS OK - 0 users currently logged in
[1416634449] External command error: Command failed

If this is a Nagios XI server, then the "SNMP Traps" service will appear as an Unconfigured object under Admin > Monitoring Config > Unconfigured Objects

 

The above notes referenced an "SNMP Traps" service. The name of the service does not have to be "SNMP Traps", this is all configurable in the EXEC line and is explained in the tutorial. The key point to make here is that a service in Nagios called "SNMP Traps" needs to exist for the status to be updated, otherwise Nagios will complain that it cannot be found.

 

However, if snmptt does NOT match an OID or MIB object identified in any of the Trap files, snmptt will log this in the file:

  • /var/log/snmptt/snmpttunknown.log

 

The key point to be made here is that while the snmptrapd service receives all traps, snmptt will only take action on the ones it has a definition for.

 

 

Spooled Traps Take A Long Time To Process

If you notice the files in the snmptt spool directory building up and it is taking a long time for them to be processed, this can be caused by slow DNS lookups.

In some implementations it has been observed that it took four hours from when the UDP traffic was observed hitting the Nagios server to when the trap was processed and Nagios received the passive check result. In this specific instance the customer was using an external DNS server (8.8.8.8 = Google public DNS server).

To stop snmptt from performing DNS lookups:

Edit the file:

/etc/snmp/snmptt.ini

 

Change the dns_enable option (line 38) so it = 0

dns_enable = 0

 

Save the file and restart the snmptt service. Once making this change you should notice the files in the snmptt spool directory being processed much quicker.

 

 

Debugging snmptt

You can enable debug logging with snmptt which can help understand what is happening with the traps being processed.

Edit the file:

/etc/snmp/snmptt.ini

Define the following settings:

DEBUGGING = 2
DEBUGGING_FILE = /var/log/snmptt/snmptt.debug

Save the file and restart the snmptt service.

Now you can observe snmptt behaviour by watching the debug log:

tail -f /var/log/snmptt/snmptt.debug
EVENT nSvcEvent .1.3.6.1.4.1.20006.1.7 "Status Events" Normal
FORMAT The SNMP trap that is generated as a result of an event with the service $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated as a result of an event with the service $*"
SDESC
The SNMP trap that is generated as a result of an event with the service
in Nagios.
Variables:
1: nHostname
2: nHostStateID
3: nSvcDesc
4: nSvcStateID
5: nSvcAttempt
6: nSvcDurationSec
7: nSvcGroupName
8: nSvcLastCheck
9: nSvcLastChange
10: nSvcOutput
EDESC - See more at: http://support.nagios.com/kb/article.php?id=77&show_category=14#sthash.QeiAnSRJ.dpuf
EVENT nSvcEvent .1.3.6.1.4.1.20006.1.7 "Status Events" Normal
FORMAT The SNMP trap that is generated as a result of an event with the service $*
EXEC /usr/local/bin/snmptraphandling.py "$r" "SNMP Traps" "$s" "$@" "$-*" "The SNMP trap that is generated as a result of an event with the service $*"
SDESC
The SNMP trap that is generated as a result of an event with the service
in Nagios.
Variables:
1: nHostname
2: nHostStateID
3: nSvcDesc
4: nSvcStateID
5: nSvcAttempt
6: nSvcDurationSec
7: nSvcGroupName
8: nSvcLastCheck
9: nSvcLastChange
10: nSvcOutput
EDESC - See more at: http://support.nagios.com/kb/article.php?id=77&show_category=14#sthash.QeiAnSRJ.dpuf

 

 

Conclusion

With these steps you will be able to confirm if the snmptt service is correctly processing SNMP Traps.

 

 

Final Thoughts

For any support related questions please visit the Nagios Support Forums at:

http://support.nagios.com/forum/

1 (1)
Article Rating (1 Votes)
Rate this article
  • Icon PDFExport to PDF
  • Icon MS-WordExport to MS Word
Attachments Attachments
There are no attachments for this article.
Related Articles RSS Feed
Nagios XI - Receiving IPv6 SNMP Traps
Viewed 7832 times since Thu, Apr 28, 2016
Nagios XI - SNMP Trap Hardening
Viewed 10721 times since Tue, Nov 6, 2018
SNMP traps are now showing UNKNOWN for the sender IP
Viewed 3648 times since Tue, Jan 19, 2021
Nagios XI - SNMP Trap v3 Configuration
Viewed 27406 times since Tue, Nov 13, 2018
SNMP Traps - Standard Handler vs Embedded Handler
Viewed 12169 times since Mon, Oct 24, 2016
SNMP Trap - Inbound UDP Traffic
Viewed 23452 times since Tue, Mar 24, 2015
SNMP Traps - Understanding Trap Variables
Viewed 19379 times since Mon, Oct 24, 2016
Nagios XI - Receiving SNMP Traps From Nagios Network Analyzer
Viewed 5800 times since Mon, Oct 17, 2016
SNMP Trap - How To Send A Test Trap
Viewed 210651 times since Fri, Apr 1, 2016
Nagios XI - How SNMP Traps Work
Viewed 5153 times since Mon, Nov 18, 2019