Page 1 of 1
snmptt not receiving alerts
Posted: Wed Aug 08, 2018 7:25 am
by hbouma
I have 6 Nagios XI servers. 5 running Nagios XI 5.5.1, 1 running 5.5.2. All 6 are on Red Hat 64bit VM's with Maria DB databases offloaded. All were originally installed as 5.4.13, then upgraded to 5.5.0, 5.5.1 and one then upgraded to 5.5.2.
2 of the servers are having an issue not receiving traps. The other 4 are working normally. Alerts are not being created, unconfigured objects are not seen when sent to Nagios XI for the first time. 1 sewrver is version 5.5.1 and the other 5.5.2. When running the One Click Systems Test from the Nagios XI GUI, it fails with a message "Failed State 4/9 (Received Trap Verification)! SNMPTT was not able to run the NXTI trap handling script, and received traps are not being added to the database".
The issue seems similar to
https://support.nagios.com/forum/viewto ... 16&t=49548, but I am having no issues with the service starting. In addition, the user in that post is getting data posted to their xi_cmp_trapdata_log table in the Nagios XI database. My table is empty.
I have verified that there are no problems with firewalls being in the way. All servers are sharing a VLAN and using the same firewall rules. I have tried sending snmptraps from the server to localhost with the error persisting. No errors are being shown in /var/log/snmptt/. I am not sure where to go from here.
Re: snmptt not receiving alerts
Posted: Wed Aug 08, 2018 9:23 am
by swolf
Since you're having the same issue on all servers, I'd like you to try these commands on the 5.5.2 system.
First, I'd like you to go to /etc/snmp/snmptt.ini and verify that net_snmp_perl_enable is set to 1. If it isn't, set it to 1 and then restart snmptt with
You may find at this point that you have the same issue as the other thread. In this case, stop here and let us know.
Otherwise, make sure snmptrapd is running with
If it's not running, start the service and run the tests again.
If this wasn't the issue, go to the Advanced tab of the SNMP Trap Interface and click the "Add Example Trap Definition" button.
Go to the Defined Traps tab and verify that an event with the name "NXTI_Event_1" exists.
Then click the 'Send Test Trap' button. In ~5-10 seconds, there should be an entry in either the Received Traps tab or the "Show Unknown Trap Log" button in the Advanced Tab. Let us know where the trap turns up (or if it doesn't make it to either location).
Re: snmptt not receiving alerts
Posted: Wed Aug 08, 2018 9:30 am
by hbouma
A restart of the snmptt service worked. It is active and running.
If this wasn't the issue, go to the Advanced tab of the SNMP Trap Interface and click the "Add Example Trap Definition" button.
Go to the Defined Traps tab and verify that an event with the name "NXTI_Event_1" exists.
Then click the 'Send Test Trap' button. In ~5-10 seconds, there should be an entry in either the Received Traps tab or the "Show Unknown Trap Log" button in the Advanced Tab. Let us know where the trap turns up (or if it doesn't make it to either location).
Example Trap Definition has NXTI_Event_1 and NXTI_Event_2.
Send Test Trap selected and sent.
Nothing showing under received traps. No unknown traps in the "Show Unknown Trap Log"
Re: snmptt not receiving alerts
Posted: Wed Aug 08, 2018 9:50 am
by swolf
Okay, let's look at the log files directly. If you're able to open multiple terminals to the XI machine, I'd like you to use one to run
which should follow changes to any of the logs in that directory.
Then, in your other terminal, run
Code: Select all
snmptrap -v 2c -c public localhost "" NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 123456
(this is all one command)
and let me know of any output from either terminal.
Also, from your last response it wasn't clear whether you checked snmptrapd as well as snmptt. If you haven't, please verify this as well:
Re: snmptt not receiving alerts
Posted: Wed Aug 08, 2018 9:58 am
by hbouma
Thank you for catching that. snmptrapd was loaded but dead.
I have restarted it. I am showing a trap in the Unknown Traps Log contents.
Tail -f is now showing the snmp traps in the snmpttunknown.log file.
I know in Nagios XI 5.4.13, we would see these alerts come in as Unconfigured Objects. Is this no longer the case? i don't see the alerts here.
Re: snmptt not receiving alerts
Posted: Wed Aug 08, 2018 1:27 pm
by hbouma
Ok, following the steps listed here
https://support.nagios.com/forum/viewto ... 74#p259174, as well as copying the missing snmptt.conf.nxti file to from another server to this one and selecting to restore the SNMPTT configuraiton, I was able to get past the failed Step 4/9.
However, I am failing at step 8/9 "Failed Stage 8/9 (Test File Truncation)! Your snmptt setup works locally, but there may be permissions issues in your /usr/local/nagiosxi/var directory."
My var directory is setup as follows:
Code: Select all
drwxr-xr-x. 7 nagios nagios 4.0K Aug 8 14:19 var
Code: Select all
drwxr-xr-x. 7 nagios nagios 4.0K Aug 8 14:19 .
drwxr-xr-x 11 nagios nagios 4.0K Jun 1 10:28 ..
drwxrwxr-x. 2 nagios nagios 4.0K Jun 29 07:45 certs
-rw-r--r-- 1 nagios nagios 3.9M Aug 8 14:19 cleaner.log
-rw-r--r-- 1 nagios nagios 25K Jul 23 03:44 cleaner.log-20180723.gz
-rw-r--r-- 1 nagios nagios 284K Aug 8 14:19 cmdsubsys.log
-rw-r--r-- 1 nagios nagios 655K Aug 6 12:35 cmdsubsys.log-20180806.gz
drwsrwsr-x. 3 apache nagios 4.0K Jun 29 14:08 components
-rw-r--r--. 1 nagios nagios 7 Aug 8 14:19 corelog.data
-rw-r--r--. 1 nagios nagios 0 Aug 8 14:19 corelog.diff
-rw-r--r-- 1 nagios nagios 3.6M Aug 8 14:15 dbmaint.log
-rw-r--r-- 1 nagios nagios 237K Jul 26 03:10 dbmaint.log-20180726.gz
-rw-r--r-- 1 nagios nagios 1.9M Aug 8 14:19 deadpool.log
-rw-r--r-- 1 nagios nagios 18K Jul 20 03:17 deadpool.log-20180720.gz
-rw-r--r-- 1 nagios nagios 0 Aug 8 14:19 event_handler.lock
-rw-r--r-- 1 nagios nagios 4.0M Aug 8 14:19 event_handler.log
-rw-r--r-- 1 nagios nagios 322K Aug 6 12:35 event_handler.log-20180806.gz
-rw-r--r-- 1 nagios nagios 6.9M Aug 8 14:19 eventman.log
-rw-r--r-- 1 nagios nagios 449K Aug 6 12:35 eventman.log-20180806.gz
-rw-r--r--. 1 nagios nagios 3.8M Aug 8 14:19 feedproc.log
drwxrwxr-x. 2 nagios nagios 4.0K Jun 29 07:45 keys
-rw-r--r--. 1 nagios nagios 998 Jun 29 07:37 load_url.log
-rw-r--r-- 1 nagios nagios 2.7M Aug 8 14:19 nom.log
-rw-r--r-- 1 nagios nagios 21K Jul 24 03:42 nom.log-20180724.gz
-rw-rw-r-- 1 root root 18 Aug 8 14:16 NXTI_Write_Test
-rw-r--r-- 1 nagios nagios 2.2M Aug 8 14:19 perfdataproc.log
-rw-r--r-- 1 nagios nagios 227K Jul 17 03:42 perfdataproc.log-20180717.gz
-rw-r--r--. 1 nagios nagios 209K Aug 8 14:01 recurringdowntime.log
-rw-r--r--. 1 nagios nagios 826K Aug 6 11:47 reportengine.log
drwxr-xr-x. 2 nagios nagios 4.0K Aug 6 14:35 subsys
-rw-r--r-- 1 nagios nagios 24M Aug 8 14:19 sysstat.log
-rw-r--r-- 1 nagios nagios 1.5M Aug 8 03:20 sysstat.log-20180808.gz
-rw-rw-r--. 1 nagios nagios 121K Jul 13 08:13 upgrade.log
drwxr-xr-x. 2 nagios nagios 4.0K Jun 1 10:28 upgrades
-rw-r--r--. 1 nagios nagios 7 Jun 1 10:34 xi-itype
-rw-r--r--. 1 nagios nagios 13K Jul 27 07:49 xi-sys.cfg
-rw-r--r--. 1 nagios nagios 37 Jun 1 10:34 xi-uuid
-rw-r--r--. 1 nagios nagios 196 Jul 27 07:49 xiversion
Re: snmptt not receiving alerts
Posted: Thu Aug 09, 2018 9:15 am
by hbouma
Issue has been corrected. The /usr/local/nagiosxi/var/NXTI_Write_Test was owned by root:root instead of nagios:nagios. It also had permissions of 664 instead of 666.
Re: snmptt not receiving alerts
Posted: Thu Aug 09, 2018 9:22 am
by cdienger
Glad to hear about the progress. Are we okay to lock this thread up?
Re: snmptt not receiving alerts
Posted: Thu Aug 09, 2018 9:30 am
by hbouma
Yes, you can lock this thread.