Stale Notifications coming in

[rafaelvdberg]

I had an issue with my Nagios XI instance last night where the ramdisc partition filled up and became full.

After resolving this issue this morning by removing the perfdata from the ramdisc partition, I have been getting email notifications all day from Nagios for outages that occurred during the Nagios outage.

Specific example, I got this email 3 minutes ago:

Notification Type: PROBLEM
Host: Pelican Grand Anevia - CDS
State: DOWN
Address: 10.255.24.170
Info: (host check orphaned, is the mod-gearman worker on queue hostgroup_monitoring running?)
Date/Time: 2015-03-05 15:58:27

This device is online and all mod_gearman workers are online now.

It seems that Nagios has somehow stored up these notifications and it slowly (all day now) sending them to me.

Looking that this device in the Nagios XI interface shows:

Host State: Up
Duration: 11h 10m 30s
State Type: Hard
Current Check: 1 of 3
Last Check: 2015-03-05 16:03:41
Next Check: 2015-03-05 16:05:15
Last State Change: 2015-03-05 04:53:19
Last Notification: Never
Check Type: Active
Check Latency: 0.01639 seconds
Execution Time: 0.31749 seconds
State Change: 0%
Performance Data: time=0.176439s;;;0.000000 size=500B;;;0

As you can see from the above, "Last State Change" was during the outage at 4:53am, however I just got the email notification 3 minutes ago.

I can provide hundreds more examples if needed.

I've checked my mail queue (using postqueue -p ) and it says "Mail queue is empty".

I'm using nagios XI 2014R1.4

How do I clear out these old notifications so I stop getting emails on them?

Thanks for your help.

Best,

Rafael

[abrist]

Does the XI notification log show these notifications getting sent around the time you are receiving them? When your ramdisk fills, you should restart all the related nagios services as they could have issues with lock files etc and may not be running correctly. (this all depends on what you moved to the ramdisk)

[rafaelvdberg]

Thank you for your reply.

No, the notification log does not show these notifications getting sent at the time I am receiving them. I think I restarted most of the Nagios services but perhaps not all.

This has stopped happening now (I guess it caught up).

[lmiltchev]

This has stopped happening now (I guess it caught up).

I'm glad the issue has been resolved! I am locking this post.