Notification History Issue

[rajasegar]

I have a service where the notifications is being sent out successfully but the history does not show anything.
Only difference compared to rest is the flapping check is disabled.

Details in my signature.

Code: Select all

define service {
	host_name			NET_INTERNET_BNG_EBB1,NET_INTERNET_BNG_EBB2
	service_description		BWC - Incoming Min Check
	use				STPT-ROUTER-BW-INTERNET
	check_command			rs_clicks_ebb_alert!$HOSTNAME$!$HOSTADDRESS$!20!20!!!!
	max_check_attempts		5
	check_interval			2
	retry_interval			1
	flap_detection_enabled		0
	notification_period		NET_EBB_BANGI_CHECK
	contacts			+infra_support
	_critical_rec_sms		true
	_critical_sms			true
	register			1
	}	

2016-07-25_11-37-17.jpg

2016-07-25_11-36-40.jpg

[Box293]

Are you seeing anything logged in /var/log/httpd/error_log when you try running the history report?

Does anything appear if you change it to a bigger period like "This Week" ?

[rajasegar]

Are you seeing anything logged in /var/log/httpd/error_log when you try running the history report?

Does anything appear if you change it to a bigger period like "This Week" ?

Nothing comes out even if I select this whole year.

No alerts when running history report. I did notice only host alerts showing up for "This month".

[tgriep]

In the Nagios XI GUI, can you go to the Admin > Performance Settings menu, click on the Databases tab and verify that the Max Notification Age is set to how you want?
By default, it should be set to 90 days.

Can you open the Event Log from the Home screen and do you see the Host or Service Notification entry there?

[rajasegar]

In the Nagios XI GUI, can you go to the Admin > Performance Settings menu, click on the Databases tab and verify that the Max Notification Age is set to how you want?
By default, it should be set to 90 days.

Can you open the Event Log from the Home screen and do you see the Host or Service Notification entry there?

Everything is fine except for this service. There is no notification history even though notifications is working fine.
The rest of the services is ok.

[tgriep]

So this one service has no information in the Notifications report and no information in the Event Log?
Can you post the configuration file for that service so we can view it?
Also login as root to the XI server, run the following command and post the output.

Code: Select all

grep NET_INTERNET_BNG_EBB2 /usr/local/nagios/var/archives/*

[rajasegar]

So this one service has no information in the Notifications report and no information in the Event Log?
Can you post the configuration file for that service so we can view it?
Also login as root to the XI server, run the following command and post the output.

Code: Select all

grep NET_INTERNET_BNG_EBB2 /usr/local/nagios/var/archives/*

Definition for service is in the first post.
There is event log just no notification history.

Code: Select all

Lots of line in the output starting from 07-24-2-16

grep NET_INTERNET_BNG_EBB2 /usr/local/nagios/var/archives/*
.... snipped
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469462740] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;CRITICAL;HARD;5;CRITICAL - NET_INTERNET_BNG_EBB2 incoming BW is 10Mbps, outgoing is 1Mbps (20/20)
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469487960] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;OK;HARD;5;OK - NET_INTERNET_BNG_EBB2 incoming BW is 20Mbps, outgoing is 5Mbps (20/20)
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469488190] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;CRITICAL;SOFT;1;CRITICAL - NET_INTERNET_BNG_EBB2 incoming BW is 18Mbps, outgoing is 4Mbps (20/20)
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469488250] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;CRITICAL;SOFT;2;CRITICAL - NET_INTERNET_BNG_EBB2 incoming BW is 18Mbps, outgoing is 4Mbps (20/20)
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469488310] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;CRITICAL;SOFT;3;CRITICAL - NET_INTERNET_BNG_EBB2 incoming BW is 18Mbps, outgoing is 4Mbps (20/20)
/usr/local/nagios/var/archives/nagios-07-27-2016-00.log:[1469488370] SERVICE ALERT: NET_INTERNET_BNG_EBB2;BWC - Incoming Min Check;OK;SOFT;4;OK - NET_INTERNET_BNG_EBB2 incoming BW is 22Mbps, outgoing is 5Mbps (20/20)

[tgriep]

Sorry about that, I meant posting the service template.
The service config you have posted, doesn't show that the notifications are enabled so we need to verify that the template has it enabled.
From the output of the grep, I don't see the notification message so it may need to be enabled.

[rajasegar]

Sorry about that, I meant posting the service template.
The service config you have posted, doesn't show that the notifications are enabled so we need to verify that the template has it enabled.
From the output of the grep, I don't see the notification message so it may need to be enabled.

Notifications are being sent out, only missing in notification history.

Code: Select all

define service {
       name                          		STPT-ROUTER-BW-INTERNET
       service_description           		STPT-ROUTER-BW-INTERNET
       use                           		STPT-0-generic-service
       max_check_attempts            		5
       check_interval                		4
       retry_interval                		1
       contact_groups                		+CGRP_INFRA_NETWORK
       register                    		0

}	

[tgriep]

Maybe it is a corrupt MYSQL entry causing it.
Do you see any errors in the /var/log/mysqld.log file?
Can you create a new copy of that check and see if that works for you?
Also, run the following as root on the server and post the output here.

Code: Select all

ps -ef --cols=300