Nagios Support Forum

Our Nagios server is in a bad state. We are unable to logon and noticed that /var was at 100%. Some of the tables under /var/lib/mysql/nagiosxi were huge and any attempt to truncate just hangs. Not sure what we can do at this point.

Thanks

Can you run the following commands, and show the output?
You may start deleting some old log files that you don't need to clear some space in /var.

What commands have you tried using to truncate the db tables?

I've tried running:

echo "truncate table xi_events; truncate xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 85G 9.0G 76G 11% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.7G 15G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 1014M 192M 823M 19% /boot
/dev/mapper/VolGroup00-LogVol03 2.0G 33M 2.0G 2% /home
/dev/mapper/VolGroup00-LogVol06 25G 33M 25G 1% /fs21
/dev/mapper/VolGroup00-LogVol07 2.0G 33M 2.0G 2% /pd_log
/dev/mapper/VolGroup00-LogVol05 30G 928M 30G 4% /opt
/dev/mapper/VolGroup00-LogVol04 40G 40G 20K 100% /var
/dev/mapper/VolGroup00-LogVol02 5.0G 38M 5.0G 1% /tmp
/dev/mapper/VolGroup00-LogVol08 40G 5.0G 35G 13% /store
tmpfs 3.2G 0 3.2G 0% /run/user/0

# du -a /var | sort -n -r | head -n 10
41808376 /var
39481420 /var/lib
39266112 /var/lib/mysql
36884432 /var/lib/mysql/nagiosxi
26192432 /var/lib/mysql/nagiosxi/xi_meta.MYD
9554252 /var/lib/mysql/nagiosxi/xi_meta.TMD
2349356 /var/lib/mysql/nagios
1980128 /var/log
1271704 /var/log/snmptt
1189320 /var/log/snmptt/snmptt.log

Truncating a large db table could take a while. It's going to take even longer, when you are trying to truncate 3 tables at once... Here's what you need to do:

1. Delete some old log files from /var/log. Make some room.
2. Make sure you have plenty of space in /tmp
3. Try truncating one table at the time. Truncating shouldn't hang your system but it may take a very long time. Be patient and again, make sure that you have some free space in /tmp and /var BEFORE running the truncate command.

Let us know if this helped.