Notifications not being sent

[JohnFLi]

I have 2 Nagios XI installations. The mostly used one works fine. THe second (PCI network vLan) isn't sending notifications. it use to, so I have no idea when it stopped working.

When I open the Nagios Core interface, and click on "Notifications" it show that nothing has been sent out for a week.
There should have been a notifications sent during this past week as there is a Linux machine in need of updates and it has been in this state for a week.

I went to teh XI interface/Admin/Manage email Settings and was able to send myself a test email. So the email settings are right. Even if it wasn't, it would have at least tried and that would be seen in the 'notification' section of the core interface.

I am using the latest version of XI

Code: Select all

Latest Available Version: 5.2.3 
Installed Version: 5.2.3 

Service Config for the item in question:

Code: Select all

define service {
 define service {
        host_name                       G1VPWEBPCIDB01
        service_description             Yum Updates
        use                             xiwizard_nrpe_service
        check_command                   check_nrpe!check_yum!!!!!!!
        max_check_attempts              5
        check_interval                  5
        retry_interval                  1
        check_period                    xi_timeperiod_24x7
        notification_interval           5
        notification_period             xi_timeperiod_24x7
        notification_options            w,c,r,
        contact_groups                  G1VPWEBPCIDB01
        _xiwizard                       linux-server
        register                        1
        }

Any ideas on where to start looking??

[Box293]

Admin > System Information > Monitoring Engine Status
Under the Monitoring Engine Process is Notifications green?

Can you please run these commands and send us the output:

Code: Select all

ps -ef | grep nagios.cfg
ipcs -q

[JohnFLi]

Yes, they are all green

ps -ef | grep nagios.cfg

Code: Select all

[root@G1VPPCINAG services]# ps -ef | grep nagios.cfg
nagios   16507     1  0 11:03 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   16523 16507  0 11:03 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root     21227 27835  0 11:19 pts/0    00:00:00 grep nagios.cfg
[root@G1VPPCINAG services]#

ipcs -q

Code: Select all

[root@G1VPPCINAG services]# ipcs -q

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0xa2000002 393216     nagios     600        0            0

[root@G1VPPCINAG services]#

[Box293]

Can you look in the file /usr/local/nagios/var/objects.cache and find the service Yum Updates for that host.

Paste it here.

Do the same for the file /usr/local/nagios/var/retention.dat

[JohnFLi]

/usr/local/nagios/var/objects.cache

Code: Select all

define service {
        host_name       G1VPWEBPCIDB01
        service_description     Yum Updates
        check_period    xi_timeperiod_24x7
        check_command   check_nrpe!check_yum!!!!!!!
        contact_groups  G1VPWEBPCIDB01
        notification_period     xi_timeperiod_24x7
        initial_state   o
        importance      0
        check_interval  5.000000
        retry_interval  1.000000
        max_check_attempts      5
        is_volatile     0
        parallelize_check       1
        active_checks_enabled   1
        passive_checks_enabled  1
        obsess  1
        event_handler_enabled   1
        low_flap_threshold      0.000000
        high_flap_threshold     0.000000
        flap_detection_enabled  1
        flap_detection_options  a
        freshness_threshold     0
        check_freshness 0
        notification_options    r,w,c
        notifications_enabled   1
        notification_interval   5.000000
        first_notification_delay        0.000000
        stalking_options        n
        process_perf_data       1
        retain_status_information       1
        retain_nonstatus_information    1
        _XIWIZARD       linux-server
        }

/usr/local/nagios/var/retention.dat

Code: Select all

service {
host_name=G1VPWEBPCIDB01
service_description=Yum Updates
modified_attributes=1
check_command=check_nrpe!check_yum!!!!!!!
check_period=xi_timeperiod_24x7
notification_period=xi_timeperiod_24x7
event_handler=
has_been_checked=1
check_execution_time=1.360
check_latency=0.000
check_type=0
current_state=1
last_state=1
last_hard_state=1
last_event_id=1940
current_event_id=1950
current_problem_id=752
last_problem_id=715
current_attempt=5
max_attempts=5
normal_check_interval=5.000000
retry_check_interval=1.000000
state_type=1
last_state_change=1450131349
last_hard_state_change=1450131349
last_time_ok=1445907649
last_time_warning=1450810829
last_time_unknown=0
last_time_critical=1450131349
plugin_output=YUM WARNING:  O/S requires an update.
long_plugin_output=
performance_data=
last_check=1450810829
next_check=1450811129
check_options=0
notified_on_unknown=0
notified_on_warning=0
notified_on_critical=0
current_notification_number=0
current_notification_id=20244
last_notification=0
notifications_enabled=1
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=1
process_performance_data=1
obsess=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
_XIWIZARD=0;linux-server
}

[Box293]

Before I go any further, does this "no notification problem" occur on all objects on this XI server?

[JohnFLi]

I just forced a different service on a different host to fail (made c drive alert at 9%) and it worked just fine

[JohnFLi]

Not sure if this is another issue, or another symptom....

When I make a change, (I see the red * ) and click on "Apply Changes"
The screen will show

Code: Select all

Waiting for configuration verification........................................................

and that's it.

If I go to TOOLS/WRITE CONFIG FILES
I can click on each of the buttons and it goes through without an issue.

[Box293]

Not sure if this is another issue, or another symptom....

When I make a change, (I see the red * ) and click on "Apply Changes"
The screen will show

Code: Select all

Waiting for configuration verification........................................................

and that's it.

If I go to TOOLS/WRITE CONFIG FILES
I can click on each of the buttons and it goes through without an issue.

Can you run the following in an ssh session on your Nagios XI host.

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Then go into XI Apply Configuration.

What output is produced in the SSH session?

Before I go any further, does this "no notification problem" occur on all objects on this XI server?

I just forced a different service on a different host to fail (made c drive alert at 9%) and it worked just fine

Can you post all the contact definitions defined in the contact group G1VPWEBPCIDB01.

[JohnFLi]

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Code: Select all

[root@G1VPPCINAG home]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
.............................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
S COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-12-22 14:03:48--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

     0K .......... .......... .......... ....                  2.40M=0.01s

2015-12-22 14:03:49 (2.40 MB/s) - “nagiosql.login” saved [34863]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=c5vnh6' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-12-22 14:03:49--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

     0K .......... .......... .......... ....                  2.41M=0.01s

2015-12-22 14:03:49 (2.41 MB/s) - “nagiosql.login” saved [34863]

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=c5vnh6' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-12-22 14:03:49--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

     0K .......... ....                                        47.8K=0.3s

2015-12-22 14:03:50 (47.8 KB/s) - “nagiosql.export.monitoring” saved [14824]

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 142 services.
        Checked 24 hosts.
        Checked 2 host groups.
        Checked 0 service groups.
        Checked 18 contacts.
        Checked 8 contact groups.
        Checked 125 commands.
        Checked 16 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 24 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 16 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0
..........
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
......................................^C
[root@G1VPPCINAG home]#

Code: Select all

define contactgroup {
        contactgroup_name                       G1VPWEBPCIDB01
        alias                                   G1VPWEBPCIDB01
        members                                 Name2,Name2_TXT,Name1,Name1_TXT
        }