SNMPTRAP delay

[questrad]

NagiosXI Version -2014R2.6
Linux- Red Hat Enterprise Linux Server release 5.11 (Tikanga)

So for the few days we have been having an issue with our snmptraps coming in later than usual. On a regular day we would get snmptrap at 5:00am to 5:10am but now they are spread from 5:00am to 6:00am.

So far this is what I have found
This email is received at 05:50:29 (suppose to be coming in around 5:10ish)
---------
***** Nagios Production Alert *****
Nagios has detected a problem with this service.

Notification Type: PROBLEM
State: CRITICAL

Host: server006 (server006.domain.COM)
(server006)

Service: IQ SNMP Traps
(IQ SNMP Traps)

Info:
someapp: (re)started

Date/Time: 2015-05-04 05:50:29
Nagios URL: https://nagios.somedomain.com/nagiosxi/
Alert sent to: support
----------

Nagios XI web interface says 05:03:08

2015-05-04 05:03:08 server006 IQ SNMP Traps Service Problem No CRITICAL support Custom: qt_xi_service_notification_handler someapp: (re)started NID: 8780541, COID: 1611, CNID: 8847861, CNMID: 8847861
----------------

Then it jumps into a rabbit hole then goes to Sendmail 47mins later

Sendmail /var/log/maillog says 5:50:30

[Mon May 04 08:57:00] root@nagios001:/# grep "t449oThY010972" /var/log/maillog
2015-05-04T05:50:29.758462-04:00 nagios001 sendmail[10972]: t449oThY010972: Authentication-Warning: nagios001.domain.com: nagios set sender to [email protected] using -f
2015-05-04T05:50:29.767982-04:00 nagios001 sendmail[10972]: t449oThY010972: from=[email protected], size=1751, class=0, nrcpts=1, msgid=<[email protected]>, relay=nagios@localhost
2015-05-04T05:50:30.676530-04:00 nagios001 sendmail[10972]: t449oThY010972: to=[email protected], ctladdr=[email protected] (5002/5002), delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=31751, relay=relay.domain.com [xxx.xxx.xxx.xxx], dsn=2.0.0, stat=Sent (<[email protected]> [InternalId=361968] Queued mail for delivery)

--------
No notification delays are setup on service/host template
I have no idea where to look next :S
Please help

[lmiltchev]

Is it possible that you had another trap coming in at around 5:50:30? Can you show us screenshots of the State History and Notifications reports for the timeperiod in question?

[questrad]

Every morning for years at 5am we get 295 snmptraps, and email notifications will come in at 5am and end at 5:10ish. But now the email notifications are spread over an hour and ending at around 6ish.

[ssax]

Do you have any escalations defined on them?

Is it only these traps that are being delayed or all notifications?

Are you having any DNS issues? NSLookup returns fast, etc?

[questrad]

Q1-Do you have any escalations defined on them?
A1-No Host or Service escalations defined.
Q2-Is it only these traps that are being delayed or all notifications?
A2- Yes it is only these traps that are being delayed.
Q3-Are you having any DNS issues? NSLookup returns fast, etc?
A3-No dns issues, nslookup is fast.

[questrad]

Could someone tell me what file/script handles the handoff of the email to sendmail? I would like to compare to healthy environment.
I just acknowledged some alerts and the notification came to me 6mins later. So it seems to be other notifications too.

[ssax]

function xicore_handle_notification_event

Code: Select all

/usr/local/nagiosxi/html/includes/components/xicore/xicore.inc.php

function send_email

Code: Select all

/usr/local/nagiosxi/html/includes/utils-email.inc.php

[questrad]

Could anyone suggest anything else?Tailing a particular log file cause I'm at a loss

[ssax]

Let's take a look at what is says in the DB for one of those delayed events:

You can select all of them and search through them like this:

Code: Select all

echo "\x \\\ select * from xi_events left join xi_meta on xi_events.event_id = xi_meta.metaobj_id;" | psql nagiosxi nagiosxi

Or you can try to limit it with something like this:

Code: Select all

echo "\x \\\ select * from xi_events left join xi_meta on xi_events.event_id = xi_meta.metaobj_id where xi_meta.keyvalue like '%YOURHOSTNAME%';" | psql nagiosxi nagiosxi

[Box293]

I suggest disabling DNS resolution for the traps being received:

The steps taken to disable DNS lookups are as follows:

snmptrapd
This requires editing the /etc/init.d/snmptrapd file
Change line 29 so it has the -n argument:

Code: Select all

OPTIONS="-n -Lsd -p /var/run/snmptrapd.pid"

Then:

Code: Select all

service restart snmptrapd

snmptt
This requires editing the /etc/snmp/snmptt.ini file
Change line 38 so it = 0

Code: Select all

dns_enable = 0

Then:

Code: Select all

service restart snmptt

Does this help?