Alerts not being Generated

[trojanholdings]

Hi All,
So I'm having a few issues with Nagios XI alert generation or the lack there of.

Normally all XI monitors will alert to email, this works fine, however I have some custom nrpe commands (Checking service states etc) which do not raise any alerts.

When i take a look at the Critical service and then go to the alert tab there have been no alerts raised. I can manually send out a notification and this works... Really not sure what is going on here.. HELP!

[rkennedy]

Do you have alerts setup for those NRPE service checks in Nagios XI? Can you post a service definition for one of the related services that isn't firing off alerts?

[trojanholdings]

Hi,
Yes i do have alerts set up, Attached is a screenshot of my failed service and the alerting settings from the CCM - Hopefully this helps!

[lmiltchev]

Can you also click on the "Manage Contacts" and "Manage Contactgroups" buttons under the "Alert Settings" tab and show us screenshots?

[trojanholdings]

Certainly

[Box293]

Can you please run these commands and post the output:

Code: Select all

grep 'Check Webpoint Process' /usr/local/nagios/var/archives/nagios-01-14-2016-*.log | perl -pe 's/(\d+)/localtime($1)/e'
grep 'Check Webpoint Process' /usr/local/nagios/var/archives/nagios-01-13-2016-*.log | perl -pe 's/(\d+)/localtime($1)/e'
grep 'Check Webpoint Process' /usr/local/nagios/var/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'

[trojanholdings]

Not a problem -

[root@monitoring ~]# grep 'Check Webpoint Process' /usr/local/nagios/var/archives/nagios-01-14-2016-*.log | perl -pe 's/(\d+)/localtime($1)/e'
[Wed Jan 13 00:00:00 2016] CURRENT SERVICE STATE: intouch-gs-us;Check Webpoint Process;OK;HARD;1;OK: All processes are running.
[Wed Jan 13 19:44:29 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;1;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:45:27 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;2;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:46:25 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;3;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:47:22 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;4;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:48:21 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;HARD;5;CRITICAL: Webpoint.exe: hung (critical)



[root@monitoring ~]# grep 'Check Webpoint Process' /usr/local/nagios/var/archives/nagios-01-13-2016-*.log | perl -pe 's/(\d+)/localtime($1)/e'
[Tue Jan 12 00:00:00 2016] CURRENT SERVICE STATE: intouch-gs-us;Check Webpoint Process;CRITICAL;HARD;5;CRITICAL: Webpoint.exe: hung (critical)
[Tue Jan 12 08:36:23 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;OK;HARD;5;OK: All processes are running.
[Tue Jan 12 18:58:00 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;1;CHECK_NRPE: Socket timeout after 30 seconds.
[Tue Jan 12 18:58:30 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;OK;SOFT;2;OK: All processes are running.



[root@monitoring ~]# grep 'Check Webpoint Process' /usr/local/nagios/var/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
[Fri Jan 15 00:00:00 2016] CURRENT SERVICE STATE: intouch-gs-us;Check Webpoint Process;OK;HARD;1;OK: All processes are running.

[Box293]

[root@monitoring ~]# grep 'Check Webpoint Process' /usr/local/nagios/var/archives/nagios-01-14-2016-*.log | perl -pe 's/(\d+)/localtime($1)/e'
[Wed Jan 13 00:00:00 2016] CURRENT SERVICE STATE: intouch-gs-us;Check Webpoint Process;OK;HARD;1;OK: All processes are running.
[Wed Jan 13 19:44:29 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;1;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:45:27 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;2;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:46:25 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;3;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:47:22 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;SOFT;4;CRITICAL: Webpoint.exe: hung (critical)
[Wed Jan 13 19:48:21 2016] SERVICE ALERT: intouch-gs-us;Check Webpoint Process;CRITICAL;HARD;5;CRITICAL: Webpoint.exe: hung (critical)

This one is good because it shows that it entered a HARD state so a notification should have been sent but there was nothing logged about it.

I can manually send out a notification and this works...

How exactly are you sending the manual notification?
Is this via the service, on the Advanced tab - "Send custom notification"

[trojanholdings]

Hi Box293

Thats exactly what I thought, I was sure that a notification would be generated, as we do get alerts for other monitors e.g. hosts down.

Yes I click on the Advanced Tab and send custom notification, Interestingly enough, if i trigger the custom notification when the service is in the Hard state, I get notification updates if the service is flapping and subsequently when the monitor recovers..


Cheers

[lmiltchev]

Are you having issues with notifications not being sent to "unstedj" contact or some other contact, member of the "xxx_Alerting_Group" contacgroup? Please, show the config of this contact.

Core Configuration Manager->Contacts->View Text Config

If this is a "xi user" (not "contact only"), log in (or masquerade) as this user, click on the username in the upper right corner, then click on "Notification Preferences" and show a screenshot of this page.