All hosts grey since reboot (again)

[tmcdonald]

What are the permissions on the retention.dat file? Make sure nagios has read+write on it.

[WillemDH]

ls -l /usr/local/nagios/var/retention.dat
-rw------- 1 nagios users 7966814 Apr 30 17:10 /usr/local/nagios/var/retention.dat

[scottwilkerson]

I have seen this a couple times and have never been able to pin it down, but the following usually fixes it...

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start

[WillemDH]

Did what you asked:

Code: Select all

 service nagios stop
Stopping nagios: .done.
[root@srvnagios01 ~]# service ndo2db stop
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
[root@srv~]# service ndo2db stop
ndo2db was not running... could not stop
[root@srv~]# service ndo2db status
ndo2db is not running
[root@srv~]# killall -9 nagios
nagios: no process killed
[root@srv~]# killall -9 ndo2db
ndo2db: no process killed
[root@srv~]# service ndo2db start
Starting ndo2db: done.
[root@srv~]# service nagios start
Starting nagios: done.

is it possible I need to stop the ndo2db service first?

Everything seems normal now, but I'm only having this issue after a reboot.

Just rebooted the server and all the hosts are still greyed out.. After applying configuration, host go green again. Even worse this time, the performance grapher service seems to not have started automatically. Had to start it manually..

[slansing]

When you rebooted, did you check to see if NDO was running? Looks like it was never started, which would cause issues for sure. You can check, when you have a rebooted server by running:

Code: Select all

service ndo2db status

[WillemDH]

Jsut did another reboot and ndo2db service was started. had to apply configuration again before hosts went green again.

[slansing]

Hmm... and MySql was running as well it looked like. Any chance we could get the following if/when you reboot again? Almost immediately after boot?:

Code: Select all

tail -30 /var/log/httpd/error_log
tail -30 /var/log/messages
tail -30 /usr/local/nagios/var/nagios.log

[WillemDH]

Hello,

I finally managed to schedule another reboot. I had the same symptoms, all hosts were greyed out. I tailed the error logs as you asked, but the logs are full of 'senesitive info, so I'l pm them to you. Please let me know what you find.

Grtz

Willem

[tmcdonald]

If you sent them to slansing, he is out of the office. Send them to me instead and I can make sure we all have access to them.

[WillemDH]

Sent them to you Trevor. Let me know what you make of them.. I'll try to clean out the warnings asap. Grtz