Ran out of Disk Monitoring Engine not running

[matt.lilek]

We ran out of disk again and now the monitoring engine is not working there is a config error that is messing nagios up. dont understand how it wrote 3gb of backup on its own to get full up and mess itself up? What can i do now?

[abrist]

You need to clear up some space, increase the disk size and most likely restart services or reboot the server. It would be nice to know where all you space went:

Code: Select all

cd /
find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
du -hsx * | sort -rh | head -10

You may want to stop nagios, npcd, ndo2db, and mysql:

Code: Select all

service nagios stop
service npcd stop
service ndo2db stop
service mysqld stop

What is you disk and inode usage right now:

Code: Select all

df -h
df -i

[matt.lilek]

sorry i should have metioned that we already increased the space to 80GB now and server is back up now it just config errors and the the monitoring engine will not start

[abrist]

You will want to restore a previous, working snapshot and then "apply configuration". Go to --> Admin --> config snapshots --> click the "restore" icon (green recycle arrows) for the topmost, working (non-red) snapshot in the list. This will restore the most recent working snapshot. Once restored, try applying the configuration.

[matt.lilek]

Just for fun i did that but the monitoring engine will not start

[matt.lilek]

sorry it did start, looks like we are good again. Just wondering why disk got full again? Folder /etc/nagiosql/backup was 3.9gb now its 86mb. It had hundred of files now it doesnt?

[lmiltchev]

I'm glad your issue has been resolved! Is "/etc/nagiosql/backup" growing up in size again? Do you see any errors in the mysqld log?

Code: Select all

tail -50 /var/log/mysqld.log