Ignore alert if too much (storm)

[steph007]

Hi,

Context
On 800+ servers I monitor NTP. NTP is crucial for us for transactional application. Only few second delay between critical servers may cause transaction failed.

Problem
In Nagios, I have one service for check_ntp where all servers are part of it.
Recently, a problem on the main time server cause problem on each of our server for NTP. So each of them report a Critical alert causing a storm in our ticketing and paging system.

Solution?
I wondering how can I avoid such situation.

[benjaminsmith]

HI,

I would take a look at Service Dependencies. Go to Configure > CCM > Advanced > Service Dependencies to set this up. It can get a little complicated but the main idea is that you can set notification behavior based on the state of the higher level service.

service-dependency.png

So for example, if the main NTP service is critical, then you would disable notifications for the dependent ntp services ( all 800+ hundred of them).

The most detailed documentation on dependencies is found on the Nagios Core documentation. A little harder to read, but very thorough.
https://assets.nagios.com/downloads/nag ... ncies.html

And our knowledgebase.
https://support.nagios.com/kb/article.php?id=505

Take a look and let me know if you need any assistance.

Regards,
Benjamin

[steph007]

Hi,

I knew that could be a option/solution, but in my context it's not.
My customer have another vendor/contractor who take care of the DNS. As a different vendor/contractor, I cannot put direct monitoring on the higher service level.

[steph007]

Sorry, NTP!

[benjaminsmith]

HI,

Using a service dependency would be the most elegant solution here, but there are other ways to control the number of notifications. If you set the notification interval to 0, it will only send one notification.

Another option is to setup Service Escalations for this check. If there was an incident you would still get the initial batch of notifications, which I assume you would want. However, you can create various levels and adjust who will be notified, how many times, and the interval. There is a wizard to set this up, just go to Configure > CCM > Tools > Escalation Wizard

Understanding Notification Escalations

[steph007]

Thanks, I will try to goes that way.
You can close the ticket, no more question.

[scottwilkerson]

Thanks, I will try to goes that way.
You can close the ticket, no more question.

Locking thread