Notifications Disabled/No Services mapped to nodes

[kotterbein]

This is a repeat of the same issue in http://support.nagios.com/forum/viewtop ... =16&t=9823

no services are showing up under individual nodes, and the notifications are showing as off/greyed out.

THe following appeared in /var/log/messages about 5 minutes after a reboot, looks like all the processes caught a sigterm:

Code: Select all

2013-03-26T09:27:19.460487-04:00 ic-nag01 ntpd[3508]: synchronized to LOCAL(0), stratum 10
2013-03-26T09:27:19.460501-04:00 ic-nag01 ntpd[3508]: kernel time sync enabled 0001
2013-03-26T09:27:26.024704-04:00 ic-nag01 nagios: SERVICE ALERT: ic-sur01b;Total CPU Util;CRITICAL;HARD;1;Percent Utilized  = 84.59 %
2013-03-26T09:27:56.096687-04:00 ic-nag01 nagios: SERVICE ALERT: ic-sur01b;Total CPU Util;WARNING;HARD;1;Percent Utilized  = 76.01 %
2013-03-26T09:28:14.279108-04:00 ic-nag01 nagios: SERVICE ALERT: ic-kix02;CPU Util;CRITICAL;HARD;1;CRITICAL - user: 60.72, nice: 0.50, sys: 6.20, iowait: 0.50, irq: 0.50, softirq: 0.50 idle: 34.57
2013-03-26T09:28:14.882439-04:00 ic-nag01 nagios: Caught SIGTERM, shutting down...
2013-03-26T09:28:15.110301-04:00 ic-nag01 nagios: Successfully shutdown... (PID=4147)
2013-03-26T09:28:15.117525-04:00 ic-nag01 nagios: Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' deinitialized successfully.
2013-03-26T09:28:15.117618-04:00 ic-nag01 nagios: ndomod: Shutdown complete.
2013-03-26T09:28:15.117694-04:00 ic-nag01 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
2013-03-26T09:28:15.992410-04:00 ic-nag01 nagios: Nagios 3.4.1 starting... (PID=6655)
2013-03-26T09:28:15.992437-04:00 ic-nag01 nagios: Local time is Tue Mar 26 09:28:15 EDT 2013
2013-03-26T09:28:15.992455-04:00 ic-nag01 nagios: LOG VERSION: 2.0
2013-03-26T09:28:15.993593-04:00 ic-nag01 nagios: mod_gearman: initialized version 1.3.6 (libgearman 0.25)
2013-03-26T09:28:15.993615-04:00 ic-nag01 nagios: Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' initialized successfully.
2013-03-26T09:28:15.993942-04:00 ic-nag01 nagios: ndomod: NDOMOD 1.5.1 (05-15-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
2013-03-26T09:28:15.994048-04:00 ic-nag01 nagios: ndomod: Successfully connected to data sink.  0 queued items to flush.
2013-03-26T09:28:15.994075-04:00 ic-nag01 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
2013-03-26T09:28:16.083659-04:00 ic-nag01 nagios: Finished daemonizing... (New PID=6660)
2013-03-26T09:28:20.125332-04:00 ic-nag01 ndo2db: Error: queue send error, retrying...
2013-03-26T09:28:24.460574-04:00 ic-nag01 ntpd[3508]: synchronized to 11.3.242.248, stratum 1

I restarted ndo2db, but still having some issues.

[kotterbein]

tail of /var/log/mysqld.log:

Code: Select all

130326  9:20:41  InnoDB: Starting shutdown...
130326  9:20:41  InnoDB: Shutdown completed; log sequence number 0 43655
130326  9:20:41 [Note] /usr/libexec/mysqld: Shutdown complete

130326 09:20:41  mysqld ended

130326 09:24:03  mysqld started
130326  9:24:04  InnoDB: Started; log sequence number 0 43655
130326  9:24:04 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

[slansing]

I noticed you said "after a reboot," how was this reboot completed? If it was a forceful reboot then you could be seeing some database corruption and I would recommend running through the database repair documentation:

http://assets.nagios.com/downloads/nagi ... tabase.pdf

Also, can you return the output of the following:

Code: Select all

service ndo2db status

service nagios status

service mysqld status

[kotterbein]

Code: Select all

service ndo2db status
ndo2db (pid 7951) is running...

Code: Select all

service nagios status
nagios (pid 6660) is running...

Code: Select all

service mysqld status
mysqld (pid 3587) is running...

reboot was done with an init 6... three times so far, but we are into production hours here so I do not want to reboot again. I ran through the DB corruption document again, and ran the repair, still with no result.

[kotterbein]

also odd-

All service checks are showing a "check pending..." however, I am getting status updates regularly. and all the performance graphs are showing an error:

"You are not authorized to access this feature. Contact your Nagios XI administrator for more information, or to obtain access to this feature."

I am logged in as nagiosadmin.

[slansing]

Can you run the following and then wait 5 minutes and check again?

Code: Select all

service nagios stop

Code: Select all

killall -9 nagios

Code: Select all

service nagios start

[kotterbein]

getting the following in /var/log/messages:

Code: Select all

2013-03-26T11:42:56.082310-04:00 ic-nag01 nagios: Nagios 3.4.1 starting... (PID=31929)
2013-03-26T11:42:56.082331-04:00 ic-nag01 nagios: Local time is Tue Mar 26 11:42:56 EDT 2013
2013-03-26T11:42:56.082376-04:00 ic-nag01 nagios: LOG VERSION: 2.0
2013-03-26T11:42:56.083587-04:00 ic-nag01 nagios: mod_gearman: initialized version 1.3.6 (libgearman 0.25)
2013-03-26T11:42:56.083611-04:00 ic-nag01 nagios: Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' initialized successfully.
2013-03-26T11:42:56.083955-04:00 ic-nag01 nagios: ndomod: NDOMOD 1.5.1 (05-15-2012) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
2013-03-26T11:42:56.084062-04:00 ic-nag01 nagios: ndomod: Successfully connected to data sink.  0 queued items to flush.
2013-03-26T11:42:56.084091-04:00 ic-nag01 nagios: Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
2013-03-26T11:42:56.175145-04:00 ic-nag01 nagios: Finished daemonizing... (New PID=31934)
2013-03-26T11:43:00.017281-04:00 ic-nag01 ndo2db: Error: queue send error, retrying...

I usually now have to restart nd02db after a nagios restart...

still no luck.

showing no hosts under the reports section either- capacity planning when attempting to run a report, dropdown host selection has no hosts- seems like nagios is not loading hosts at all-

host status summary is empty as well, yet on the "host detail" I see all the hosts listed and checks running for host status.

[abrist]

OP opened a ticket. Locking thread as email support will deal with the issues.