Nagios doesn't call notification

[jweijters]

Hi,

We see a starnge problem in our Nagios

We have seen several hosts with services going to WARNING or CRITICAL STATE HARD but the notificantion isn't triggerd.
As you can see below, the host has a service gone in WARNING, however aftre the MAX 3 times the notification does't run
Can you please help me? I created a system profile to upload.

For instance this host:

Screenshot 2021-11-15 at 14.21.18.png

here is the State history:

Screenshot 2021-11-15 at 14.23.30.png

And the notification history:

Screenshot 2021-11-15 at 14.24.02.png

Kind regards,

Joris Weijters

[kfanselow]

Hi Joris,

What version of core are you currently using ?

Code: Select all

 /usr/local/nagios/bin/nagios -h  | head -n 4 

Please feel free to PM me the system profile and we'd be happy to take a look.

Thanks and Best Regards,
Keith

[jweijters]

Hi Kieth,

We are using Nagios 5.8.6,

Code: Select all

$ /usr/local/nagios/bin/nagios -h  | head -n 4

Nagios Core 4.4.6
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad

I PM the system profile to you.

Kind regards,

Joris Weijters

[kfanselow]

Hi Joris,

Thank you for sending the profile - you have a rather complex environment. After review one of our engineers recommended you try the following:

1) Add a user with an email address you can access for test purposes.

2) Set the user's Host and Service Notification Period to xi_timeperiod_24x7
(Configure {top} -> Core Config Manager -> Contacts under Alerting {left} -> Alert Settings tab )

3) Add the user to the cg_security_ms-k2

4) Trigger the Expiring Certificates alert again.

Also it appear that the php information was not generated - could you provide the output from the following command for us:

Code: Select all

  php -v 

Thanks and Best Regards,
Keith

[jweijters]

Hi Kieth,

Hereby I send the php -v

Code: Select all

[root@sbhppsssm008 /home/[email protected]]$ php -v
PHP 7.4.25 (cli) (built: Oct 19 2021 15:18:10) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.25, Copyright (c), by Zend Technologies
[root@sbhppsssm008 /home/[email protected]]$

I added myself to the contactgroup cg_security_ms-k2 and used "Submit passive check result" to Send an OK to this event.
Then I did a force imediate check 3 times. You can see these actions in the State History Log below.

Screenshot 2021-11-17 at 07.47.47.png

I didn't recieve an Email. there is no Notification in the notification log

Screenshot 2021-11-17 at 07.51.04.png

I will send you a new system profile

[jweijters]

Hi,

I Think I found the problem, being the contact group wich was directly connected to the host.
Does a directly connected contactgroup overrule contactgroups connected connection via the templates?
After I removed the direct connected contactgroup, I reissued the check, and I did get a notification.


Kind regards,

Joris Weijters

[jweijters]

For the AVR host, we can see the Notifications again.

However for host zena-ena-p-exmgt we still have this problem although there is no customergroup directly connected to this host.
see below.

Screenshot 2021-11-17 at 15.21.43.png

Screenshot 2021-11-17 at 15.22.15.png

Screenshot 2021-11-17 at 15.22.38.png

Can you please look in to this further

Kind regards,

Joris Weijters

[kfanselow]

Hi Joris,

Looking at the first host you referenced we noticed that the users in the contact group had a service and host notification period defined as tp_none, which is empty:

Code: Select all

define timeperiod {
        timeperiod_name tp_none
        alias   leeg
        }

This is why we suggested you create a user with host and service notification periods defined as xi_timeperiod_24x7 and add them to the contact group yesterday - to see if the system generated a notification for a user inside the defined notification time period.

For reference here is the definition of xi_timeperiod_24x:

Code: Select all

define timeperiod {
        timeperiod_name xi_timeperiod_24x7
        alias   24x7
        sunday  00:00-24:00
        monday  00:00-24:00
        tuesday 00:00-24:00
        wednesday       00:00-24:00
        thursday        00:00-24:00
        friday  00:00-24:00
        saturday        00:00-24:00
        }

This second host, zena-ena-p-exmgt, has a different contact group defined however all the members of that group also have tp_none defined for their notification period as well ( no notification window defined for the members of the group ).

The link below explains how time periods are defined and used by Nagios including for notifications:

https://assets.nagios.com/downloads/nag ... riods.html

Also for reference here's a quick way to get a look at the contact_groups associated with your hosts from the command line:

Code: Select all

 grep -A6 "define host "  /usr/local/nagios/var/objects.cache  | less 

As well as your Contact Group definitions:

Code: Select all

 grep -B1 -A4 contactgroup_name /usr/local/nagios/var/objects.cache |less  

And the notification periods for your contacts:

Code: Select all

 grep -A4 "define contact " /usr/local/nagios/var/objects.cache | less  

Thanks and Best Regards,
Keith

[jweijters]

Hi Kieth,

As we place the object.cache at the ramdisk, I issued this at the ramdisk.
we never connect a contactgroup directly to a host. the contact groups for alerting are connected to a host template
cg_alert_p1 is connected to ht_alert_sw3-7x24

My host zena-ena-p-exmgt is also connected to host template ht_alert_sw3-7x24
So why is my host then not connected to the contactgroup?

It seems there is a dependency on the order of host templates in the host definition
If my order is like this

Screenshot 2021-11-18 at 12.48.16.png

my host looks like this in the objects.cache

Code: Select all

--
define host {
        host_name zena-ena-p-exmgt
        alias Exchange Management
        address 172.17.9.6
        check_command check_ping!500,50%!700,70% -t 30!!!!!!
        contact_groups cg_security_ms-k1
        notification_period tp_7x24
--

If my order is like this:

Screenshot 2021-11-18 at 12.52.04.png

my host looks like this:

Code: Select all

---
define host {
        host_name       zena-ena-p-exmgt
        alias   Exchange Management
        address 172.17.9.6
        check_command   check_ping!500,50%!700,70% -t 30!!!!!!
        contact_groups  cg_alert_p1,cg_security_ms-k1
        notification_period     tp_7x24
--

Can you please explain this to me?

Kind regards,

Joris Weijters

[kfanselow]

Hi Joris,

The quick answer is Additive Inheritance.

The cg_alert_p1 contact group in the host template ht_alert_sw3-7x24 was defined with Additive Inheritance but the cg_security_ms-k1 group in ht_customer_zena was not. Below are the template definitions from your generated hosttemplates.cfg file, note the contact_groups line and the + symbol on cg_alert_p1:

Code: Select all

define host {
    name                          ht_alert_sw3-7x24
    alias                         servicewindow 7x24
    contact_groups                +cg_alert_p1
    notification_interval         0
    notification_period           tp_7x24
    first_notification_delay      0
    notification_options          d,
    notifications_enabled         1
    icon_image                    ../../../../custom-includes/images/24-16px.png
    _SW                           7x24
    register                      0
}

Code: Select all

define host {
    name                          ht_customer_zena
    alias                         REDACTED
    contact_groups                cg_security_ms-k1
    _cpu_crit                     95
    _cpu_warn                     90
    _customer                     REDACTED
    _disk_crit                    98
    _disk_query                   -q FixedDisk -t 30 -G -S0,1 -m ''
    _disk_warn                    95
    _mem_crit                     100
    _mem_warn                     98
    _snmp_community               REDACTED
    register                      0
}

You can set Additive Inheritance for your contact group by selecting the + in the Inheritance options at the bottom of the Manage Contact Groups window on the Host Template Management page ( see attached screenshot ).

Below is a link to our documentation that explains how object inheritance works with templates:

https://assets.nagios.com/downloads/nag ... tance.html

Hope this is useful.

Thanks and Best Regards,
Keith