Delayed alert notifications??

[kane3030]

Hello,

I am having a problem where, intermittently, alerts are being detected but sending out hours later. For example:

Notification Type: PROBLEM

Service: ****
Host: ****
Address: ****
State: CRITICAL

Date/Time: Fri Aug 2 18:21:20 EDT 2013

Additional Info:

Aug 2 13:24:15


The issue happened at 13:24 but was first being sent as a notification at 18:21.

[abrist]

This sounds like a timezone or date/time issue. What is the output of:

Code: Select all

grep date.timezone /etc/php.ini
date

[kane3030]

I will paste the results of that shortly but the issue is very intermittent. It works most of the time and will randomly send out alerts late. If you look at what I sent, its a five hour difference but its not always five hours, its some random time.

[abrist]

Are these problems happening for all checks, or a specific subset? Checks like snmp traps, if there are dns issues, may stack up and come in late.

[kane3030]

This is happening for all checks, regardless of host. Just happened again for an alert:


Date/Time: Tue Aug 6 15:41:16 EDT 2013

Additional Info:

Aug 5 13:17:14

[sreinhardt]

Let's check the snmptt spool.

Code: Select all

 ls -lv /var/spool/snmptt/

[kane3030]

I actually don't have that under /var/spool. I didn't originally set up this Nagios installation but now I am the admin of it. I restarted snmpd this morning in hopes that it may be the cause but this has been causing a lot of stress for my co workers.

[abrist]

Lets us know. We still have not heard from you concerning the server date, nor the current request for your snmptt spool directory listing.

[kane3030]

After trying many different things, it appears restarting the snmpd process worked. Thanks for all of your suggestions!

[abrist]

The snmptt spooler may have been backed up. Causing some results to not be checked until well after they were received, effectively sending out late alerts. I have seen it before. Let us know if this recurs.