;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
;)
snmptrapd Service
The snmptrapd service is what receives an SNMP Trap from the networking stack and then places it into a spool directory for the snmptt service to process.
These steps help confirm that the snmptrapd service is working correctly. Before proceeding you would have followed the other troubleshooting articles "Inbound UDP Traffic" and "Firewall Rules".
Editing Files
In many steps of this article you will be required to edit files. This documentation will use the vi text editor. When using the vi
editor:
- To make changes press i on the keyboard first to enter insert mode
- Press Esc to exit insert mode
- When you have finished, save the changes in vi by typing :wq and press Enter
Is The snmptrapd Service Running?
Establish an SSH session to the Nagios server that is receiving SNMP Traps and execute the command for your operating system (OS):
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl status snmptrapd.service
If snmptrapd is running, it should produce output like:
snmptrapd (pid 11612) is running...
If snmptrapd is NOT running, it will produce output like:
snmptrapd is stopped
If the snmptrapd service is NOT running, this means that spooled SNMP Traps will not be rejected. To ENABLE snmptrapd on boot and to start it, execute the following commands depending on your OS:
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl enable snmptrapd.service
systemctl start snmptrapd.service
If snmptrapd is NOT installed, it will produce output like:
snmptrapd: unrecognized service
If you find that snmptrapd is not installed, you need to follow the guide "How to Integrate SNMP Traps With Nagios XI".
Also, you should confirm that the snmptt service is installed, if it is not then the snmptrapd service is going to have problems. Execute the command for your operating system (OS):
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl status snmptt.service
If snmptt is running, it should produce output like:
snmptt (pid 11612) is running...
If snmptt is NOT running, it will produce output like:
snmptt is stopped
If you find that snmptt is not installed, you need to follow the guide "How to Integrate SNMP Traps With Nagios XI".
Restart snmptrapd Service
Steps below will require you to restart the snmptrapd service, the commands for your OS are as follows:
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl restart snmptrapd.service
Stop snmptrapd Service
Steps below will require you to stop the snmptrapd service, the commands for your OS are as follows:
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl stop snmptrapd.service
Confirm Traps Are Spooled
When SNMP Traps are working correctly they are placed into the spool folder by snmptrapd and almost instantly snmptt processes the spooled trap and then deletes it. In the following steps, we are going to stop the snmptt service so it does not process the trap. This lets you confirm that the received traps are actually being spooled.
To stop the snmptt service execute the command for your operating system (OS):
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl stop snmptt.service
Which should produce the output like:
Stopping snmptt: [ OK ]
Next, from the device that sends SNMP Traps, get it to send through a trap.
Once received, it will be created as a file in /var/spool/snmptt/
To see if the file exists type the following command:
ls -al /var/spool/snmptt/
Which should show a directory listing of the spooled traps, for example they are named:
-rw-r--r-- 1 root root 395 Mar 25 11:43 #snmptt-trap-1427244215674243
You can view the contents of the trap with the following command:
cat /var/spool/snmptt/#snmptt-trap-1427244215674243
Which should output something like:
1427244215
snmpsender.domain.local
UDP: [10.25.5.20]:50655->[10.25.5.30]
DISMAN-EVENT-MIB::sysUpTimeInstance 0:1:10:08.85
SNMPv2-MIB::snmpTrapOID.0 SNMPv2-SMI::enterprises.20006.1.7
SNMPv2-SMI::enterprises.20006.1.3.1.2 "CentOS"
SNMPv2-SMI::enterprises.20006.1.3.1.6 "Users"
SNMPv2-SMI::enterprises.20006.1.3.1.7 0
SNMPv2-SMI::enterprises.20006.1.3.1.17 "USERS OK - 0 users currently logged in"
The first line is the number 1427244215, this is the time stamp of when the trap was received (epoch value).
Files Created in /var/spool/snmptt/
If you are seeing files being created in the directory /var/spool/snmptt/, then this confirms that snmptrap is correctly working.
Files NOT being created in /var/spool/snmptt/
If you are NOT seeing files being created in the directory /var/spool/snmptt/, then there may be a permissions issue. To check the permissions execute the command:
ls -dl /var/spool/snmptt/
Which should show the permissions as follows:
drwxrwxr-x 2 snmptt snmptt 4096 Mar 25 11:51 /var/spool/snmptt/
If the permissions and owner are not correct then execute these commands:
chown snmptt:snmptt /var/spool/snmptt/
chmod u+rwx,g+rwx,o+rx /var/spool/snmptt/
Then confirm the permissions are now correct:
ls -dl /var/spool/snmptt/
Once you have done this, from the device that sends SNMP Traps, get it to send through a trap. Then confirm it is created as a file in /var/spool/snmptt/.
If you are still NOT seeing files being created in the directory /var/spool/snmptt/, then there may be an issue with the snmptrapd configuration. To see the configuration execute the following command:
cat /etc/snmp/snmptrapd.conf
This shows the snmptrapd configuration file. This is a very basic file and should look like this:
disableAuthorization yes
traphandle default /usr/sbin/snmptthandler
If your snmptrapd.conf is different, please correct it (using an editor like "vi") and then restart the snmptrapd service.
You should also confirm that the following file exists and is at least version 1.2 by executing the following command:
/usr/sbin/snmptthandler --version
Which would produce this output:
SNMPTTHANDLER v1.2
(c) 2002-2007 Alex Burger
If you are still not receiving SNMP traps in the snmptt spool directory, please confirm the spool directory setting used by executing the following command:
grep spool_directory /etc/snmp/snmptt.ini
Which should produce this output:
spool_directory = /var/spool/snmptt/
Please confirm this directory exists AND the permissions are correct (covered in an earlier section in this article).
Another problem which can delay SNMP traps arriving in the snmptt spool directory can be caused by slow DNS lookups.
In some implementations it has been observed that it took four hours from when the UDP traffic was observed hitting the Nagios server to when the trap file was created in the spool directory. In this specific instance the customer was using an external DNS server (8.8.8.8 = Google public DNS server).
To stop snmptrapd from performing DNS lookups edit the file:
/etc/init.d/snmptrapd
Change the OPTIONS (line 29) so it has the -n argument:
OPTIONS="-n -Lsd -p /var/run/snmptrapd.pid"
Save the file and restart the snmptrapd service.
If you are still having problems, your next step is to enable logging.
snmptrapd Logging
These steps explain how to enable the snmptrapd daemon to logging it's output to a file. This is useful for determining exactly what snmptrapd is doing with the SNMP Traps it receives.
To enable the option you need to edit the INIT script to add an extra option. The steps are slightly different depending on the operating system version being used:
RHEL 6 | CentOS 6 | Oracle Linux 6
Edit the /etc/sysconfig/snmptrapd file:
Find this line:
OPTIONS="-Ln -p /var/run/snmptrapd.pid"
Add -Lf /var/log/snmptrapd.log to the line:
OPTIONS="-Ln -Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid"
Now restart the SNMPTRAPD service:
service snmptrapd restart
You can now proceed to the Check Log File section below.
RHEL 7 | CentOS 7 | Oracle Linux 7
Edit the /etc/sysconfig/snmptrapd file:
Add this line:
OPTIONS="-Lsd -Lf /var/log/snmptrapd.log"
Now restart the SNMPTRAPD service:
systemctl daemon-reload
systemctl restart snmptrapd.service
You can now proceed to the Check Log File section below.
Ubuntu 16
Edit the /etc/default/snmptrapd file:
Find this line:
TRAPDOPTS='-Lsd -p /run/snmptrapd.pid'
Add -Lf /var/log/snmptrapd.log to the line:
TRAPDOPTS='-Lsd -Lf /var/log/snmptrapd.log -p /run/snmptrapd.pid'
Now restart the SNMPTRAPD service:
systemctl daemon-reload
systemctl restart snmptrapd.service
You can now proceed to the Check Log File section below.
Debian 9 | Ubuntu 18
Execute the following command:
systemctl edit snmptrapd.service
Paste the following into the new file:
[Service]
ExecStart=
ExecStart=/usr/sbin/snmptrapd -Ln -f -Lf /var/log/snmptrapd.log
Now restart the SNMPTRAPD service:
systemctl daemon-reload
systemctl restart snmptrapd.service
You can now proceed to the Check Log File section below.
Check Log File
This adds a line to the snmptrapd.conf file to enable logging. The snmptrapd activity is now logged in :
/var/log/snmptrapd.log
You can watch the log information by running this command:
tail -f /var/log/snmptrapd.log
Which should product output like:
NET-SNMP version 5.7.2
2015-03-25 13:25:45 snmpsender.box293.local [UDP: [10.25.5.20]:53145->[10.25.5.30]]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1033824) 2:52:18.24 SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.20006.1.7 SNMPv2-SMI::enterprises.20006.1.3.1.2 = STRING: "CentOS" SNMPv2-SMI::enterprises.20006.1.3.1.6 = STRING: "Users" SNMPv2-SMI::enterprises.20006.1.3.1.7 = INTEGER: 1SNMPv2-SMI::enterprises.20006.1.3.1.17 = STRING: "USERS WARNING - 1 users currently logged in"
snmptrapd Debug Output
These steps explain how to start the snmptrapd daemon to log more verbose debug output to the screen. This is useful for determining exactly what snmptrapd is doing with the SNMP Traps it receives.
First you need to stop the snmptrapd service. by executing the command for your OS.
Now execute the following command to start snmptrapd in the console:
snmptrapd -D -f -Loe
Any traps received will now be output on the screen. You can use this output to help troubleshoot snmptrapd issues you may be having.
When you have finished, press CTRL + C to stop snmptrapd and then start the service so it resumes normal operation.
Start snmptt Service
Once you've completed troubleshooting make sure you start the snmptt service again with the following command for your operating system (OS):
RHEL 7+ | CentOS 7+ | Oracle Linux 7+ | Debian | Ubuntu 16/18/20
systemctl start snmptt.service
Then confirm the spooled files have been processed with the following command:
ls -al /var/spool/snmptt/
Which should show an empty directory listing.
Conclusion
With these steps you will be able to confirm if the snmptrapd service is correctly receiving SNMP Traps from a remote server.
Your next troubleshooting step would be to refer to the snmptt Service troubleshooting.
Final Thoughts
For any support related questions please visit the Nagios Support Forums at:
Subscribe to Article
Print Article
Email Article to Friend
Export to PDF
Export to MS Word
