Page 1 of 1
Ignore alert if too much (storm)
Posted: Wed Oct 14, 2020 8:21 am
by steph007
Hi,
Context
On 800+ servers I monitor NTP. NTP is crucial for us for transactional application. Only few second delay between critical servers may cause transaction failed.
Problem
In Nagios, I have one service for check_ntp where all servers are part of it.
Recently, a problem on the main time server cause problem on each of our server for NTP. So each of them report a Critical alert causing a storm in our ticketing and paging system.
Solution?
I wondering how can I avoid such situation.
Re: Ignore alert if too much (storm)
Posted: Wed Oct 14, 2020 5:21 pm
by benjaminsmith
HI,
I would take a look at Service Dependencies. Go to Configure > CCM > Advanced > Service Dependencies to set this up. It can get a little complicated but the main idea is that you can set notification behavior based on the state of the higher level service.
service-dependency.png
So for example, if the main NTP service is critical, then you would disable notifications for the dependent ntp services ( all 800+ hundred of them).
The most detailed documentation on dependencies is found on the Nagios Core documentation. A little harder to read, but very thorough.
https://assets.nagios.com/downloads/nag ... ncies.html
And our knowledgebase.
https://support.nagios.com/kb/article.php?id=505
Take a look and let me know if you need any assistance.
Regards,
Benjamin
Re: Ignore alert if too much (storm)
Posted: Thu Oct 15, 2020 8:10 am
by steph007
Hi,
I knew that could be a option/solution, but in my context it's not.
My customer have another vendor/contractor who take care of the DNS. As a different vendor/contractor, I cannot put direct monitoring on the higher service level.
Re: Ignore alert if too much (storm)
Posted: Thu Oct 15, 2020 8:50 am
by steph007
Sorry, NTP!
Re: Ignore alert if too much (storm)
Posted: Thu Oct 15, 2020 1:32 pm
by benjaminsmith
HI,
Using a service dependency would be the most elegant solution here, but there are other ways to control the number of notifications. If you set the notification interval to 0, it will only send one notification.
Another option is to setup Service Escalations for this check. If there was an incident you would still get the initial batch of notifications, which I assume you would want. However, you can create various levels and adjust who will be notified, how many times, and the interval. There is a wizard to set this up, just go to Configure > CCM > Tools > Escalation Wizard
Understanding Notification Escalations
Re: Ignore alert if too much (storm)
Posted: Fri Oct 16, 2020 8:14 am
by steph007
Thanks, I will try to goes that way.
You can close the ticket, no more question.
Re: Ignore alert if too much (storm)
Posted: Fri Oct 16, 2020 8:25 am
by scottwilkerson
steph007 wrote:Thanks, I will try to goes that way.
You can close the ticket, no more question.
Locking thread