Page 1 of 2
Notifications not being sent
Posted: Tue Dec 22, 2015 1:36 pm
by JohnFLi
I have 2 Nagios XI installations. The mostly used one works fine. THe second (PCI network vLan) isn't sending notifications. it use to, so I have no idea when it stopped working.
When I open the Nagios Core interface, and click on "Notifications" it show that nothing has been sent out for a week.
There should have been a notifications sent during this past week as there is a Linux machine in need of updates and it has been in this state for a week.
I went to teh XI interface/Admin/Manage email Settings and was able to send myself a test email. So the email settings are right. Even if it wasn't, it would have at least tried and that would be seen in the 'notification' section of the core interface.
I am using the latest version of XI
Code: Select all
Latest Available Version: 5.2.3
Installed Version: 5.2.3
Service Config for the item in question:
Code: Select all
define service {
define service {
host_name G1VPWEBPCIDB01
service_description Yum Updates
use xiwizard_nrpe_service
check_command check_nrpe!check_yum!!!!!!!
max_check_attempts 5
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 5
notification_period xi_timeperiod_24x7
notification_options w,c,r,
contact_groups G1VPWEBPCIDB01
_xiwizard linux-server
register 1
}
Any ideas on where to start looking??
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 2:27 pm
by Box293
Admin > System Information > Monitoring Engine Status
Under the Monitoring Engine Process is Notifications green?
Can you please run these commands and send us the output:
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 2:36 pm
by JohnFLi
Yes, they are all green
ps -ef | grep nagios.cfg
Code: Select all
[root@G1VPPCINAG services]# ps -ef | grep nagios.cfg
nagios 16507 1 0 11:03 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios 16523 16507 0 11:03 ? 00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root 21227 27835 0 11:19 pts/0 00:00:00 grep nagios.cfg
[root@G1VPPCINAG services]#
ipcs -q
Code: Select all
[root@G1VPPCINAG services]# ipcs -q
------ Message Queues --------
key msqid owner perms used-bytes messages
0xa2000002 393216 nagios 600 0 0
[root@G1VPPCINAG services]#
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 2:39 pm
by Box293
Can you look in the file /usr/local/nagios/var/objects.cache and find the service Yum Updates for that host.
Paste it here.
Do the same for the file /usr/local/nagios/var/retention.dat
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 2:58 pm
by JohnFLi
/usr/local/nagios/var/objects.cache
Code: Select all
define service {
host_name G1VPWEBPCIDB01
service_description Yum Updates
check_period xi_timeperiod_24x7
check_command check_nrpe!check_yum!!!!!!!
contact_groups G1VPWEBPCIDB01
notification_period xi_timeperiod_24x7
initial_state o
importance 0
check_interval 5.000000
retry_interval 1.000000
max_check_attempts 5
is_volatile 0
parallelize_check 1
active_checks_enabled 1
passive_checks_enabled 1
obsess 1
event_handler_enabled 1
low_flap_threshold 0.000000
high_flap_threshold 0.000000
flap_detection_enabled 1
flap_detection_options a
freshness_threshold 0
check_freshness 0
notification_options r,w,c
notifications_enabled 1
notification_interval 5.000000
first_notification_delay 0.000000
stalking_options n
process_perf_data 1
retain_status_information 1
retain_nonstatus_information 1
_XIWIZARD linux-server
}
/usr/local/nagios/var/retention.dat
Code: Select all
service {
host_name=G1VPWEBPCIDB01
service_description=Yum Updates
modified_attributes=1
check_command=check_nrpe!check_yum!!!!!!!
check_period=xi_timeperiod_24x7
notification_period=xi_timeperiod_24x7
event_handler=
has_been_checked=1
check_execution_time=1.360
check_latency=0.000
check_type=0
current_state=1
last_state=1
last_hard_state=1
last_event_id=1940
current_event_id=1950
current_problem_id=752
last_problem_id=715
current_attempt=5
max_attempts=5
normal_check_interval=5.000000
retry_check_interval=1.000000
state_type=1
last_state_change=1450131349
last_hard_state_change=1450131349
last_time_ok=1445907649
last_time_warning=1450810829
last_time_unknown=0
last_time_critical=1450131349
plugin_output=YUM WARNING: O/S requires an update.
long_plugin_output=
performance_data=
last_check=1450810829
next_check=1450811129
check_options=0
notified_on_unknown=0
notified_on_warning=0
notified_on_critical=0
current_notification_number=0
current_notification_id=20244
last_notification=0
notifications_enabled=1
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=1
process_performance_data=1
obsess=1
is_flapping=0
percent_state_change=0.00
check_flapping_recovery_notification=0
state_history=1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1
_XIWIZARD=0;linux-server
}
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 3:12 pm
by Box293
Before I go any further, does this "no notification problem" occur on all objects on this XI server?
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 4:09 pm
by JohnFLi
I just forced a different service on a different host to fail (made c drive alert at 9%) and it worked just fine
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 4:53 pm
by JohnFLi
Not sure if this is another issue, or another symptom....
When I make a change, (I see the red * ) and click on "Apply Changes"
The screen will show
Code: Select all
Waiting for configuration verification........................................................
and that's it.
If I go to TOOLS/WRITE CONFIG FILES
I can click on each of the buttons and it goes through without an issue.
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 5:00 pm
by Box293
JohnFLi wrote:Not sure if this is another issue, or another symptom....
When I make a change, (I see the red * ) and click on "Apply Changes"
The screen will show
Code: Select all
Waiting for configuration verification........................................................
and that's it.
If I go to TOOLS/WRITE CONFIG FILES
I can click on each of the buttons and it goes through without an issue.
Can you run the following in an ssh session on your Nagios XI host.
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Then go into XI Apply Configuration.
What output is produced in the SSH session?
Box293 wrote:Before I go any further, does this "no notification problem" occur on all objects on this XI server?
JohnFLi wrote:I just forced a different service on a different host to fail (made c drive alert at 9%) and it worked just fine
Can you post all the contact definitions defined in the contact group G1VPWEBPCIDB01.
Re: Notifications not being sent
Posted: Tue Dec 22, 2015 5:08 pm
by JohnFLi
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Code: Select all
[root@G1VPPCINAG home]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
.............................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
S COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-12-22 14:03:48-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”
0K .......... .......... .......... .... 2.40M=0.01s
2015-12-22 14:03:49 (2.40 MB/s) - “nagiosql.login” saved [34863]
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=c5vnh6' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-12-22 14:03:49-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”
0K .......... .......... .......... .... 2.41M=0.01s
2015-12-22 14:03:49 (2.41 MB/s) - “nagiosql.login” saved [34863]
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=c5vnh6' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-12-22 14:03:49-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”
0K .......... .... 47.8K=0.3s
2015-12-22 14:03:50 (47.8 KB/s) - “nagiosql.export.monitoring” saved [14824]
WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL
Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
Read object config files okay...
Running pre-flight check on configuration data...
Checking objects...
Checked 142 services.
Checked 24 hosts.
Checked 2 host groups.
Checked 0 service groups.
Checked 18 contacts.
Checked 8 contact groups.
Checked 125 commands.
Checked 16 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 24 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 16 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...
Total Warnings: 0
Total Errors: 0
Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0
..........
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
......................................^C
[root@G1VPPCINAG home]#
Code: Select all
define contactgroup {
contactgroup_name G1VPWEBPCIDB01
alias G1VPWEBPCIDB01
members Name2,Name2_TXT,Name1,Name1_TXT
}