Host notifications disabled displayed incorrectly

[justinm]

I'm running into a weird problem. The default view (home page) I use for service issues have the following filters "Filters: Service=Warning,Unknown,Critical,Not Acknowledged,Not In Downtime " This view for some reason always shows the host as grey and the notifications disabled icon. However when you click on a host or goto a different view it says the host is green and notifications are enabled. I have checked for MySQL corruption and rebooted the server however it still is shows up wrong. I don't think this happened during my last update. It appears to have happened after a normal adding of a host. I did it with the clone wizard so nothing major should have changed. System details below.

I'm using the following URL (host name not included)
https://(mynagioshost)/nagiosxi/includes/components/xicore/status.php?show=services&servicestatustypes=28&serviceattr=10

Nagios XI Version : 2012R2.7
lo-mon01 2.6.32-431.20.3.el6.x86_64 x86_64
CentOS release 6.5 (Final)
Gnome is not installed
nagios (pid 20577) is running...
NPCD running (pid 1860).
ndo2db (pid 20643) is running...
CPU Load 15: 0.19
Total Hosts: 488
Total Services: 1584

[lmiltchev]

Click on the "Process Info" and show us a screenshot of this page. Also run the following commands and show us the output in code wraps:

Code: Select all

ps -ef | grep bin/nagios
tail -20 /var/log/messages
tail -20 /var/log/httpd/error_log
tail -20 /var/log/mysqld.log
tail -20 /usr/local/nagios/var/nagios.log

[justinm]

Process info shows all green and enabled and run time of 7h 43M 57s.

[root@lo-mon01 ~]# ps -ef | grep bin/nagios
nagios 2338 1 0 Aug03 ? 00:03:51 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
[root@lo-mon01 ~]# tail-20 /var/log/messages
Aug 4 06:47:59 lo-mon01 nagios: SERVICE ALERT: exch;cpu;OK;SOFT;2;OK CPU Load ok.
Aug 4 06:49:49 lo-mon01 nagios: SERVICE NOTIFICATION: user1;util2;automatic_services;CRITICAL;xi_service_notification_handler;CRITICAL: CASUniversalAgent: stopped (critical)
Aug 4 06:54:09 lo-mon01 nagios: SERVICE NOTIFICATION: user1;ad02;disks;WARNING;notify-service-by-email;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
Aug 4 06:54:09 lo-mon01 nagios: SERVICE NOTIFICATION: user1;ad02;disks;WARNING;xi_service_notification_handler;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
Aug 4 06:59:49 lo-mon01 nagios: SERVICE NOTIFICATION: user1;util2;automatic_services;CRITICAL;xi_service_notification_handler;CRITICAL: CASUniversalAgent: stopped (critical)
Aug 4 07:02:39 lo-mon01 nagios: SERVICE ALERT: ptk-bi;cpu;WARNING;SOFT;1;WARNING: 30s: average load 81% > warning
Aug 4 07:02:59 lo-mon01 nagios: SERVICE ALERT: utility2;cpu;CRITICAL;SOFT;1;CRITICAL: 1m: average load 90% > critical, 30s: average load 99% > critical
Aug 4 07:03:39 lo-mon01 nagios: SERVICE ALERT: ptk-bi;cpu;OK;SOFT;2;OK CPU Load ok.
Aug 4 07:03:39 lo-mon01 nagios: SERVICE ALERT: irsa.dmz;memory;CRITICAL;SOFT;1;CRITICAL: physical memory: Total: 4G - Used: 3.91G (97%) - Free: 96.1M (3%) > critical, virtual memory: Total: 8T - Used: 66.4M (0%) - Free: 8T (100%), paged bytes: Total: 8G - Used: 2.31G (28%) - Free: 5.69G (72%), page file: Total: 8G - Used: 2.31G (28%) - Free: 5.69G (72%)
Aug 4 07:03:49 lo-mon01 nagios: SERVICE ALERT: econnect5;cpu;CRITICAL;SOFT;1;CRITICAL: 1m: average load 88% > warning, 30s: average load 96% > critical
Aug 4 07:03:59 lo-mon01 nagios: SERVICE ALERT: utility2;cpu;CRITICAL;SOFT;2;CRITICAL: 1m: average load 93% > critical, 30s: average load 90% > critical
Aug 4 07:04:09 lo-mon01 nagios: SERVICE NOTIFICATION: user1;ad02;disks;WARNING;notify-service-by-email;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
Aug 4 07:04:09 lo-mon01 nagios: SERVICE NOTIFICATION: user1;ad02;disks;WARNING;xi_service_notification_handler;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
Aug 4 07:04:39 lo-mon01 nagios: SERVICE ALERT: irsa.dmz;memory;OK;SOFT;2;OK: physical memory: Total: 4G - Used: 2.71G (67%) - Free: 1.29G (33%), virtual memory: Total: 8T - Used: 66.4M (0%) - Free: 8T (100%), paged bytes: Total: 8G - Used: 2.27G (28%) - Free: 5.73G (72%), page file: Total: 8G - Used: 2.27G (28%) - Free: 5.73G (72%)
Aug 4 07:04:49 lo-mon01 nagios: SERVICE ALERT: econnect5;cpu;OK;SOFT;2;OK CPU Load ok.
Aug 4 07:04:59 lo-mon01 nagios: SERVICE ALERT: utility2;cpu;WARNING;HARD;3;WARNING: 1m: average load 84% > warning, 30s: average load 82% > warning
Aug 4 07:04:59 lo-mon01 nagios: SERVICE ALERT: econnect6;cpu;CRITICAL;SOFT;1;CRITICAL: 30s: average load 93% > critical
Aug 4 07:05:09 lo-mon01 nagios: SERVICE ALERT: gisnt1;cpu;WARNING;SOFT;1;WARNING: 1m: average load 80% > warning
Aug 4 07:05:59 lo-mon01 nagios: SERVICE ALERT: econnect6;cpu;OK;SOFT;2;OK CPU Load ok.
Aug 4 07:06:09 lo-mon01 nagios: SERVICE ALERT: gisnt1;cpu;OK;SOFT;2;OK CPU Load ok.
[root@lo-mon01 ~]# tail -20 /var/log/httpd/error_log
[Sun Aug 03 03:11:01 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 03 03:11:01 2014] [notice] Digest: done
[Sun Aug 03 03:11:01 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Aug 03 19:40:46 2014] [notice] caught SIGTERM, shutting down
[Sun Aug 03 20:47:40 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Aug 03 20:47:40 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 03 20:47:40 2014] [notice] Digest: done
[Sun Aug 03 20:47:42 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Aug 03 21:00:10 2014] [notice] caught SIGTERM, shutting down
[Sun Aug 03 21:22:45 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Aug 03 21:22:45 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 03 21:22:45 2014] [notice] Digest: done
[Sun Aug 03 21:22:47 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[Sun Aug 03 21:23:07 2014] [notice] caught SIGTERM, shutting down
[Sun Aug 03 23:33:19 2014] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sun Aug 03 23:33:19 2014] [notice] Digest: generating secret for digest authentication ...
[Sun Aug 03 23:33:19 2014] [notice] Digest: done
[Sun Aug 03 23:33:21 2014] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips configured -- resuming normal operations
[root@lo-mon01 ~]# tail -20 /var/log/mysqld.log
140803 21:22:41 InnoDB: Completed initialization of buffer pool
140803 21:22:41 InnoDB: Started; log sequence number 0 44233
140803 21:22:41 [Note] Event Scheduler: Loaded 0 events
140803 21:22:41 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
140803 21:23:08 [Note] /usr/libexec/mysqld: Normal shutdown

140803 21:23:08 [Note] Event Scheduler: Purging the queue. 0 events
140803 21:23:10 InnoDB: Starting shutdown...
140803 21:23:12 InnoDB: Shutdown completed; log sequence number 0 44233
140803 21:23:12 [Note] /usr/libexec/mysqld: Shutdown complete

140803 21:23:12 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140803 23:33:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140803 23:33:15 InnoDB: Initializing buffer pool, size = 8.0M
140803 23:33:15 InnoDB: Completed initialization of buffer pool
140803 23:33:15 InnoDB: Started; log sequence number 0 44233
140803 23:33:15 [Note] Event Scheduler: Loaded 0 events
140803 23:33:15 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
[root@lo-mon01 ~]# tail -20 /usr/local/nagios/var/nagios.log
[1407152879] SERVICE ALERT: exch;cpu;OK;SOFT;2;OK CPU Load ok.
[1407152989] SERVICE NOTIFICATION: user1;util2;automatic_services;CRITICAL;xi_service_notification_handler;CRITICAL: CASUniversalAgent: stopped (critical)
[1407153249] SERVICE NOTIFICATION: user1;ad02;disks;WARNING;notify-service-by-email;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
[1407153249] SERVICE NOTIFICATION: user1;ad02;disks;WARNING;xi_service_notification_handler;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
[1407153589] SERVICE NOTIFICATION: user1;util2;automatic_services;CRITICAL;xi_service_notification_handler;CRITICAL: CASUniversalAgent: stopped (critical)
[1407153759] SERVICE ALERT: ptk-bi;cpu;WARNING;SOFT;1;WARNING: 30s: average load 81% > warning
[1407153779] SERVICE ALERT: utility2;cpu;CRITICAL;SOFT;1;CRITICAL: 1m: average load 90% > critical, 30s: average load 99% > critical
[1407153819] SERVICE ALERT: ptk-bi;cpu;OK;SOFT;2;OK CPU Load ok.
[1407153819] SERVICE ALERT: irsa;memory;CRITICAL;SOFT;1;CRITICAL: physical memory: Total: 4G - Used: 3.91G (97%) - Free: 96.1M (3%) > critical, virtual memory: Total: 8T - Used: 66.4M (0%) - Free: 8T (100%), paged bytes: Total: 8G - Used: 2.31G (28%) - Free: 5.69G (72%), page file: Total: 8G - Used: 2.31G (28%) - Free: 5.69G (72%)
[1407153829] SERVICE ALERT: econnect5;cpu;CRITICAL;SOFT;1;CRITICAL: 1m: average load 88% > warning, 30s: average load 96% > critical
[1407153839] SERVICE ALERT: utility2;cpu;CRITICAL;SOFT;2;CRITICAL: 1m: average load 93% > critical, 30s: average load 90% > critical
[1407153849] SERVICE NOTIFICATION: user1;gv-ad02;disks;WARNING;notify-service-by-email;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
[1407153849] SERVICE NOTIFICATION: user1;gv-ad02;disks;WARNING;xi_service_notification_handler;WARNING: C:\: Total: 79.9G - Used: 72.3G (90%) - Free: 7.62G (10%) warning
[1407153879] SERVICE ALERT: ed-irsa.dmz;memory;OK;SOFT;2;OK: physical memory: Total: 4G - Used: 2.71G (67%) - Free: 1.29G (33%), virtual memory: Total: 8T - Used: 66.4M (0%) - Free: 8T (100%), paged bytes: Total: 8G - Used: 2.27G (28%) - Free: 5.73G (72%), page file: Total: 8G - Used: 2.27G (28%) - Free: 5.73G (72%)
[1407153889] SERVICE ALERT: econnect5;cpu;OK;SOFT;2;OK CPU Load ok.
[1407153899] SERVICE ALERT: utility2;cpu;WARNING;HARD;3;WARNING: 1m: average load 84% > warning, 30s: average load 82% > warning
[1407153899] SERVICE ALERT: econnect6;cpu;CRITICAL;SOFT;1;CRITICAL: 30s: average load 93% > critical
[1407153909] SERVICE ALERT: gisnt1;cpu;WARNING;SOFT;1;WARNING: 1m: average load 80% > warning
[1407153959] SERVICE ALERT: econnect6;cpu;OK;SOFT;2;OK CPU Load ok.
[1407153969] SERVICE ALERT: gisnt1;cpu;OK;SOFT;2;OK CPU Load ok.

[scottwilkerson]

I've usually came across this when something happens to ndo2db, can you run

Code: Select all

service ndo2db stop
service nagios stop
killall -9 ndo2db
killall-9 nagios
service ndo2db start
service nagios start

reporting any errors