Page 1 of 1
Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 10:27 am
by matt.lilek
We ran out of disk again and now the monitoring engine is not working there is a config error that is messing nagios up. dont understand how it wrote 3gb of backup on its own to get full up and mess itself up? What can i do now?
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 10:56 am
by abrist
You need to clear up some space, increase the disk size and most likely restart services or reboot the server. It would be nice to know where all you space went:
Code: Select all
cd /
find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
du -hsx * | sort -rh | head -10
You may want to stop nagios, npcd, ndo2db, and mysql:
Code: Select all
service nagios stop
service npcd stop
service ndo2db stop
service mysqld stop
What is you disk and inode usage right now:
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 11:25 am
by matt.lilek
sorry i should have metioned that we already increased the space to 80GB now and server is back up now it just config errors and the the monitoring engine will not start
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 11:37 am
by abrist
You will want to restore a previous, working snapshot and then "apply configuration". Go to --> Admin --> config snapshots --> click the "restore" icon (green recycle arrows) for the topmost, working (non-red) snapshot in the list. This will restore the most recent working snapshot. Once restored, try applying the configuration.
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 1:52 pm
by matt.lilek
Just for fun i did that but the monitoring engine will not start
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 2:06 pm
by matt.lilek
sorry it did start, looks like we are good again. Just wondering why disk got full again? Folder /etc/nagiosql/backup was 3.9gb now its 86mb. It had hundred of files now it doesnt?
Re: Ran out of Disk Monitoring Engine not running
Posted: Mon Dec 16, 2013 2:18 pm
by lmiltchev
I'm glad your issue has been resolved! Is "/etc/nagiosql/backup" growing up in size again? Do you see any errors in the mysqld log?