Notifications are disabled

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Notifications are disabled

Post by tonyleatwork »

Hi -

My notifications are reported as being disabled. I noticed that when making certain changes in the CCM, it may come back and it may go away again. I've tried to back out all of my changes (last changes being additional users through CCM) but it's still 'stuck'.

Nagios Log
[root@nwd2ng01 var]# tail -40 nagios.log
[1411592555] ndomod registered for aggregated status data'
[1411592555] ndomod registered for retention data'
[1411592555] ndomod registered for contact data'
[1411592555] ndomod registered for contact notification data'
[1411592555] ndomod registered for acknowledgement data'
[1411592555] ndomod registered for state change data'
[1411592555] ndomod registered for contact status data'
[1411592555] ndomod registered for adaptive contact data'
[1411592555] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1411592555] Warning: Duplicate definition found for service 'Memory Usage' on host 'Nwd2spcbdev2_WMI' (config file '/usr/local/nagios/etc/services/Windows WMI DEV QA Group Memory Usage.cfg', starting on line 16)
[1411592555] Warning: Host 'drbw4.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'drcrm6.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'drerp3.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'drprd3.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'drptl4.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'drsap9.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2app6.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2bw1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2crm4.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2dri.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2erp1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ora1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ora2.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ora3.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ora4.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ora6.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2pft1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2pft2' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2prd1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2prd3corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2ptl4.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2sbx1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2sm3.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2tws1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2wms1.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2wms2.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Warning: Host 'nwd2wmsd.corp.analog.com' has no default contacts or contactgroups defined!
[1411592555] Successfully launched command file worker with pid 25976
[1411592669] SERVICE ALERT: Nwd2spcbdev3_WMI;CPU Usage;UNKNOWN;SOFT;1;UNKNOWN - Plugin Timed out (15 sec)
[1411592675] SERVICE NOTIFICATION: nagiosadmin;Nwd2epmdev1_WMI;Drive C: Disk Usage;WARNING;xi_service_notification_handler;WARNING - [Triggered by _Used%>80] - C: Total=19.99GB, Used=16.06GB (80.3%), Free=3.94GB (19.7%)
[root@nwd2ng01 var]#
Thanks.
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Notifications are disabled

Post by lmiltchev »

It seems like some of your hosts don't have contacts or contact groups defined. I would recommend assigning contacts or a contact groups in the CCM to these hosts, in order to fix the issue.
Next, run the Write Config Tool prior to applying configuration to check for config errors. Once you don't have any config errors, you can apply configuration.

Please, read our documentation on how to configure email and text notifications here:

http://assets.nagios.com/downloads/nagi ... ios-XI.pdf

Let us know if you run into more issues.
Be sure to check out our Knowledgebase for helpful articles and solutions!
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: Notifications are disabled

Post by abrist »

The log above shows a notification going out after the service restart. Are you noticing disabled notifications directly on objects, or program wide?
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
tmcdonald
Posts: 9117
Joined: Mon Sep 23, 2013 8:40 am

Re: Notifications are disabled

Post by tmcdonald »

Also, are your notifications actually disabled or does it just say they are? Do you still get email/sms alerts?

What does the following command reveal?

Code: Select all

grep "enable_notifications" /usr/local/nagios/var/status.dat
Former Nagios employee
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Notifications are disabled

Post by tonyleatwork »

lmiltchev wrote:It seems like some of your hosts don't have contacts or contact groups defined. I would recommend assigning contacts or a contact groups in the CCM to these hosts, in order to fix the issue.
Next, run the Write Config Tool prior to applying configuration to check for config errors. Once you don't have any config errors, you can apply configuration.

Please, read our documentation on how to configure email and text notifications here:

http://assets.nagios.com/downloads/nagi ... ios-XI.pdf

Let us know if you run into more issues.
That's correct, only some hosts are enabled for notifications directly.

The other hosts are in host groups with services defined against the group. I've also created a host template with specific alert criteria and applied it to the group as well. What I'm trying to achieve is to configure notifications group-wide instead of per host.
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Notifications are disabled

Post by tonyleatwork »

abrist wrote:The log above shows a notification going out after the service restart. Are you noticing disabled notifications directly on objects, or program wide?
The notifications seem to be working but slowly, the indication I got that it is program wide are the 3 exclamation marks in the Nagios XI console. When clicking on one of those and forcing the monitoring engine to run, it's still stuck.
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Notifications are disabled

Post by tonyleatwork »

tmcdonald wrote:Also, are your notifications actually disabled or does it just say they are? Do you still get email/sms alerts?

What does the following command reveal?

Code: Select all

grep "enable_notifications" /usr/local/nagios/var/status.dat
Notifications are not disabled, I did notice some email alerts came through.

grep enabled_notifications /usr/local/nagios/var/status.dat

shows nothing, but notifications_enabled grep showed a ton:

enable_notifications=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
notifications_enabled=1
no_more_notifications=0
(truncated)
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Notifications are disabled

Post by tonyleatwork »

Update:

Today, the notifications look OK again. It wasn't OK for at least an hour before I left work yesterday. The only abnormal behavior I noticed is that ndo2db is running at high cpu.. could this be the cause? It's consistently running at 100% and has been for a while. Reboot attempts have not done anything.

It is on a VM with 4 cpus and 8GB of ram, san storage back end.

25935 nagios 20 0 51744 3236 1008 R 100.0 0.0 1061:55 ndo2db
tonyleatwork
Posts: 91
Joined: Mon Jul 07, 2014 8:55 am

Re: Notifications are disabled

Post by tonyleatwork »

Update:

I tried to add a notification contact to a host template to apply to a host and the same issue is happening again. The "Notifications are disabled" blue exclamation mark along with "Active Service Checks" and "Host Service Checks" in the Nagios XI console.
abrist
Red Shirt
Posts: 8334
Joined: Thu Nov 15, 2012 1:20 pm

Re: Notifications are disabled

Post by abrist »

What version of XI are you running?
Are any of your db tables crashed?

Code: Select all

grep crashed /var/log/mysqld.log
How big are your tables?

Code: Select all

ls -lahS /var/lib/mysql/nagios | head -10
Former Nagios employee
"It is turtles. All. The. Way. Down. . . .and maybe an elephant or two."
VI VI VI - The editor of the Beast!
Come to the Dark Side.
Locked