Nagios Support Forum

NagiosXI Version -2014R2.6
Linux- Red Hat Enterprise Linux Server release 5.11 (Tikanga)

So for the few days we have been having an issue with our snmptraps coming in later than usual. On a regular day we would get snmptrap at 5:00am to 5:10am but now they are spread from 5:00am to 6:00am.

So far this is what I have found
This email is received at 05:50:29 (suppose to be coming in around 5:10ish)
---------
***** Nagios Production Alert *****
Nagios has detected a problem with this service.

Notification Type: PROBLEM
State: CRITICAL

Host: server006 (server006.domain.COM)
(server006)

Service: IQ SNMP Traps
(IQ SNMP Traps)

Info:
someapp: (re)started

Date/Time: 2015-05-04 05:50:29
Nagios URL: https://nagios.somedomain.com/nagiosxi/
Alert sent to: support
----------

Nagios XI web interface says 05:03:08

2015-05-04 05:03:08 server006 IQ SNMP Traps Service Problem No CRITICAL support Custom: qt_xi_service_notification_handler someapp: (re)started NID: 8780541, COID: 1611, CNID: 8847861, CNMID: 8847861
----------------

Then it jumps into a rabbit hole then goes to Sendmail 47mins later

Sendmail /var/log/maillog says 5:50:30

[Mon May 04 08:57:00] root@nagios001:/# grep "t449oThY010972" /var/log/maillog
2015-05-04T05:50:29.758462-04:00 nagios001 sendmail[10972]: t449oThY010972: Authentication-Warning: nagios001.domain.com: nagios set sender to [email protected] using -f
2015-05-04T05:50:29.767982-04:00 nagios001 sendmail[10972]: t449oThY010972: from=[email protected], size=1751, class=0, nrcpts=1, msgid=<[email protected]>, relay=nagios@localhost
2015-05-04T05:50:30.676530-04:00 nagios001 sendmail[10972]: t449oThY010972: to=[email protected], ctladdr=[email protected] (5002/5002), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=31751, relay=relay.domain.com [xxx.xxx.xxx.xxx], dsn=2.0.0, stat=Sent (<[email protected]> [InternalId=361968] Queued mail for delivery)

--------
No notification delays are setup on service/host template
I have no idea where to look next :S
Please help

Is it possible that you had another trap coming in at around 5:50:30? Can you show us screenshots of the State History and Notifications reports for the timeperiod in question?

Every morning for years at 5am we get 295 snmptraps, and email notifications will come in at 5am and end at 5:10ish. But now the email notifications are spread over an hour and ending at around 6ish.

Do you have any escalations defined on them?

Is it only these traps that are being delayed or all notifications?

Are you having any DNS issues? NSLookup returns fast, etc?

Q1-Do you have any escalations defined on them?
A1-No Host or Service escalations defined.
Q2-Is it only these traps that are being delayed or all notifications?
A2- Yes it is only these traps that are being delayed.
Q3-Are you having any DNS issues? NSLookup returns fast, etc?
A3-No dns issues, nslookup is fast.

Could someone tell me what file/script handles the handoff of the email to sendmail? I would like to compare to healthy environment.
I just acknowledged some alerts and the notification came to me 6mins later. So it seems to be other notifications too.

function xicore_handle_notification_event
function send_email

Could anyone suggest anything else?Tailing a particular log file cause I'm at a loss

Let's take a look at what is says in the DB for one of those delayed events:

You can select all of them and search through them like this:
Or you can try to limit it with something like this:

I suggest disabling DNS resolution for the traps being received:

The steps taken to disable DNS lookups are as follows:

snmptrapd
This requires editing the /etc/init.d/snmptrapd file
Change line 29 so it has the -n argument: Then: snmptt
This requires editing the /etc/snmp/snmptt.ini file
Change line 38 so it = 0 Then: Does this help?