Page 2 of 4
Re: All hosts grey since reboot (again)
Posted: Wed Apr 30, 2014 10:10 am
by tmcdonald
What are the permissions on the retention.dat file? Make sure nagios has read+write on it.
Re: All hosts grey since reboot (again)
Posted: Wed Apr 30, 2014 10:31 am
by WillemDH
ls -l /usr/local/nagios/var/retention.dat
-rw------- 1 nagios users 7966814 Apr 30 17:10 /usr/local/nagios/var/retention.dat
Re: All hosts grey since reboot (again)
Posted: Wed Apr 30, 2014 10:42 am
by scottwilkerson
I have seen this a couple times and have never been able to pin it down, but the following usually fixes it...
Code: Select all
service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start
Re: All hosts grey since reboot (again)
Posted: Wed Apr 30, 2014 1:04 pm
by WillemDH
Did what you asked:
Code: Select all
service nagios stop
Stopping nagios: .done.
[root@srvnagios01 ~]# service ndo2db stop
Stopping ndo2db: head: cannot open `/usr/local/nagios/var/ndo2db.lock' for reading: No such file or directory
done.
[root@srv~]# service ndo2db stop
ndo2db was not running... could not stop
[root@srv~]# service ndo2db status
ndo2db is not running
[root@srv~]# killall -9 nagios
nagios: no process killed
[root@srv~]# killall -9 ndo2db
ndo2db: no process killed
[root@srv~]# service ndo2db start
Starting ndo2db: done.
[root@srv~]# service nagios start
Starting nagios: done.
is it possible I need to stop the ndo2db service first?
Everything seems normal now, but I'm only having this issue after a reboot.
Just rebooted the server and all the hosts are still greyed out..
After applying configuration, host go green again. Even worse this time, the performance grapher service seems to not have started automatically. Had to start it manually..
Re: All hosts grey since reboot (again)
Posted: Wed Apr 30, 2014 3:09 pm
by slansing
When you rebooted, did you check to see if NDO was running? Looks like it was never started, which would cause issues for sure. You can check, when you have a rebooted server by running:
Re: All hosts grey since reboot (again)
Posted: Tue May 06, 2014 5:49 am
by WillemDH
Jsut did another reboot and ndo2db service was started. had to apply configuration again before hosts went green again.
Re: All hosts grey since reboot (again)
Posted: Tue May 06, 2014 12:13 pm
by slansing
Hmm... and MySql was running as well it looked like. Any chance we could get the following if/when you reboot again? Almost immediately after boot?:
Code: Select all
tail -30 /var/log/httpd/error_log
tail -30 /var/log/messages
tail -30 /usr/local/nagios/var/nagios.log
Re: All hosts grey since reboot (again)
Posted: Wed May 28, 2014 5:03 am
by WillemDH
Hello,
I finally managed to schedule another reboot. I had the same symptoms, all hosts were greyed out. I tailed the error logs as you asked, but the logs are full of 'senesitive info, so I'l pm them to you. Please let me know what you find.
Grtz
Willem
Re: All hosts grey since reboot (again)
Posted: Wed May 28, 2014 9:18 am
by tmcdonald
If you sent them to slansing, he is out of the office. Send them to me instead and I can make sure we all have access to them.
Re: All hosts grey since reboot (again)
Posted: Wed May 28, 2014 10:08 am
by WillemDH
Sent them to you Trevor. Let me know what you make of them.. I'll try to clean out the warnings asap. Grtz