snmptt not receiving alerts

[hbouma]

I have 6 Nagios XI servers. 5 running Nagios XI 5.5.1, 1 running 5.5.2. All 6 are on Red Hat 64bit VM's with Maria DB databases offloaded. All were originally installed as 5.4.13, then upgraded to 5.5.0, 5.5.1 and one then upgraded to 5.5.2.

2 of the servers are having an issue not receiving traps. The other 4 are working normally. Alerts are not being created, unconfigured objects are not seen when sent to Nagios XI for the first time. 1 sewrver is version 5.5.1 and the other 5.5.2. When running the One Click Systems Test from the Nagios XI GUI, it fails with a message "Failed State 4/9 (Received Trap Verification)! SNMPTT was not able to run the NXTI trap handling script, and received traps are not being added to the database".

The issue seems similar to https://support.nagios.com/forum/viewto ... 16&t=49548, but I am having no issues with the service starting. In addition, the user in that post is getting data posted to their xi_cmp_trapdata_log table in the Nagios XI database. My table is empty.

I have verified that there are no problems with firewalls being in the way. All servers are sharing a VLAN and using the same firewall rules. I have tried sending snmptraps from the server to localhost with the error persisting. No errors are being shown in /var/log/snmptt/. I am not sure where to go from here.

[swolf]

Since you're having the same issue on all servers, I'd like you to try these commands on the 5.5.2 system.

First, I'd like you to go to /etc/snmp/snmptt.ini and verify that net_snmp_perl_enable is set to 1. If it isn't, set it to 1 and then restart snmptt with

Code: Select all

service snmptt restart

You may find at this point that you have the same issue as the other thread. In this case, stop here and let us know.

Otherwise, make sure snmptrapd is running with

Code: Select all

service snmptrapd status

If it's not running, start the service and run the tests again.

If this wasn't the issue, go to the Advanced tab of the SNMP Trap Interface and click the "Add Example Trap Definition" button.
Go to the Defined Traps tab and verify that an event with the name "NXTI_Event_1" exists.
Then click the 'Send Test Trap' button. In ~5-10 seconds, there should be an entry in either the Received Traps tab or the "Show Unknown Trap Log" button in the Advanced Tab. Let us know where the trap turns up (or if it doesn't make it to either location).

[hbouma]

A restart of the snmptt service worked. It is active and running.

If this wasn't the issue, go to the Advanced tab of the SNMP Trap Interface and click the "Add Example Trap Definition" button.
Go to the Defined Traps tab and verify that an event with the name "NXTI_Event_1" exists.
Then click the 'Send Test Trap' button. In ~5-10 seconds, there should be an entry in either the Received Traps tab or the "Show Unknown Trap Log" button in the Advanced Tab. Let us know where the trap turns up (or if it doesn't make it to either location).

Example Trap Definition has NXTI_Event_1 and NXTI_Event_2.
Send Test Trap selected and sent.
Nothing showing under received traps. No unknown traps in the "Show Unknown Trap Log"

[swolf]

Okay, let's look at the log files directly. If you're able to open multiple terminals to the XI machine, I'd like you to use one to run

Code: Select all

cd /var/log/snmptt
tail -f *

which should follow changes to any of the logs in that directory.
Then, in your other terminal, run

Code: Select all

snmptrap -v 2c -c public localhost "" NET-SNMP-EXAMPLES-MIB::netSnmpExampleHeartbeatNotification netSnmpExampleHeartbeatRate i 123456

(this is all one command)
and let me know of any output from either terminal.

Also, from your last response it wasn't clear whether you checked snmptrapd as well as snmptt. If you haven't, please verify this as well:

Code: Select all

service snmptrapd status

[hbouma]

Thank you for catching that. snmptrapd was loaded but dead.

I have restarted it. I am showing a trap in the Unknown Traps Log contents.

Tail -f is now showing the snmp traps in the snmpttunknown.log file.

I know in Nagios XI 5.4.13, we would see these alerts come in as Unconfigured Objects. Is this no longer the case? i don't see the alerts here.

[hbouma]

Ok, following the steps listed herehttps://support.nagios.com/forum/viewto ... 74#p259174, as well as copying the missing snmptt.conf.nxti file to from another server to this one and selecting to restore the SNMPTT configuraiton, I was able to get past the failed Step 4/9.

However, I am failing at step 8/9 "Failed Stage 8/9 (Test File Truncation)! Your snmptt setup works locally, but there may be permissions issues in your /usr/local/nagiosxi/var directory."

My var directory is setup as follows:

Code: Select all

drwxr-xr-x.  7 nagios nagios 4.0K Aug  8 14:19 var

Code: Select all

drwxr-xr-x.  7 nagios nagios 4.0K Aug  8 14:19 .
drwxr-xr-x  11 nagios nagios 4.0K Jun  1 10:28 ..
drwxrwxr-x.  2 nagios nagios 4.0K Jun 29 07:45 certs
-rw-r--r--   1 nagios nagios 3.9M Aug  8 14:19 cleaner.log
-rw-r--r--   1 nagios nagios  25K Jul 23 03:44 cleaner.log-20180723.gz
-rw-r--r--   1 nagios nagios 284K Aug  8 14:19 cmdsubsys.log
-rw-r--r--   1 nagios nagios 655K Aug  6 12:35 cmdsubsys.log-20180806.gz
drwsrwsr-x.  3 apache nagios 4.0K Jun 29 14:08 components
-rw-r--r--.  1 nagios nagios    7 Aug  8 14:19 corelog.data
-rw-r--r--.  1 nagios nagios    0 Aug  8 14:19 corelog.diff
-rw-r--r--   1 nagios nagios 3.6M Aug  8 14:15 dbmaint.log
-rw-r--r--   1 nagios nagios 237K Jul 26 03:10 dbmaint.log-20180726.gz
-rw-r--r--   1 nagios nagios 1.9M Aug  8 14:19 deadpool.log
-rw-r--r--   1 nagios nagios  18K Jul 20 03:17 deadpool.log-20180720.gz
-rw-r--r--   1 nagios nagios    0 Aug  8 14:19 event_handler.lock
-rw-r--r--   1 nagios nagios 4.0M Aug  8 14:19 event_handler.log
-rw-r--r--   1 nagios nagios 322K Aug  6 12:35 event_handler.log-20180806.gz
-rw-r--r--   1 nagios nagios 6.9M Aug  8 14:19 eventman.log
-rw-r--r--   1 nagios nagios 449K Aug  6 12:35 eventman.log-20180806.gz
-rw-r--r--.  1 nagios nagios 3.8M Aug  8 14:19 feedproc.log
drwxrwxr-x.  2 nagios nagios 4.0K Jun 29 07:45 keys
-rw-r--r--.  1 nagios nagios  998 Jun 29 07:37 load_url.log
-rw-r--r--   1 nagios nagios 2.7M Aug  8 14:19 nom.log
-rw-r--r--   1 nagios nagios  21K Jul 24 03:42 nom.log-20180724.gz
-rw-rw-r--   1 root   root     18 Aug  8 14:16 NXTI_Write_Test
-rw-r--r--   1 nagios nagios 2.2M Aug  8 14:19 perfdataproc.log
-rw-r--r--   1 nagios nagios 227K Jul 17 03:42 perfdataproc.log-20180717.gz
-rw-r--r--.  1 nagios nagios 209K Aug  8 14:01 recurringdowntime.log
-rw-r--r--.  1 nagios nagios 826K Aug  6 11:47 reportengine.log
drwxr-xr-x.  2 nagios nagios 4.0K Aug  6 14:35 subsys
-rw-r--r--   1 nagios nagios  24M Aug  8 14:19 sysstat.log
-rw-r--r--   1 nagios nagios 1.5M Aug  8 03:20 sysstat.log-20180808.gz
-rw-rw-r--.  1 nagios nagios 121K Jul 13 08:13 upgrade.log
drwxr-xr-x.  2 nagios nagios 4.0K Jun  1 10:28 upgrades
-rw-r--r--.  1 nagios nagios    7 Jun  1 10:34 xi-itype
-rw-r--r--.  1 nagios nagios  13K Jul 27 07:49 xi-sys.cfg
-rw-r--r--.  1 nagios nagios   37 Jun  1 10:34 xi-uuid
-rw-r--r--.  1 nagios nagios  196 Jul 27 07:49 xiversion

[hbouma]

Issue has been corrected. The /usr/local/nagiosxi/var/NXTI_Write_Test was owned by root:root instead of nagios:nagios. It also had permissions of 664 instead of 666.

[cdienger]

Glad to hear about the progress. Are we okay to lock this thread up?

[hbouma]

Yes, you can lock this thread.