Nagios in bad state.

[ksafa]

Our Nagios server is in a bad state. We are unable to logon and noticed that /var was at 100%. Some of the tables under /var/lib/mysql/nagiosxi were huge and any attempt to truncate just hangs. Not sure what we can do at this point.

Thanks

[lmiltchev]

Can you run the following commands, and show the output?

Code: Select all

df -h
du -a /var | sort -n -r | head -n 10

You may start deleting some old log files that you don't need to clear some space in /var.

What commands have you tried using to truncate the db tables?

[ksafa]

I've tried running:

echo "truncate table xi_events; truncate xi_meta; truncate table xi_eventqueue;" | mysql -u root -pnagiosxi nagiosxi

#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 85G 9.0G 76G 11% /
devtmpfs 16G 0 16G 0% /dev
tmpfs 16G 0 16G 0% /dev/shm
tmpfs 16G 1.7G 15G 11% /run
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda1 1014M 192M 823M 19% /boot
/dev/mapper/VolGroup00-LogVol03 2.0G 33M 2.0G 2% /home
/dev/mapper/VolGroup00-LogVol06 25G 33M 25G 1% /fs21
/dev/mapper/VolGroup00-LogVol07 2.0G 33M 2.0G 2% /pd_log
/dev/mapper/VolGroup00-LogVol05 30G 928M 30G 4% /opt
/dev/mapper/VolGroup00-LogVol04 40G 40G 20K 100% /var
/dev/mapper/VolGroup00-LogVol02 5.0G 38M 5.0G 1% /tmp
/dev/mapper/VolGroup00-LogVol08 40G 5.0G 35G 13% /store
tmpfs 3.2G 0 3.2G 0% /run/user/0

# du -a /var | sort -n -r | head -n 10
41808376 /var
39481420 /var/lib
39266112 /var/lib/mysql
36884432 /var/lib/mysql/nagiosxi
26192432 /var/lib/mysql/nagiosxi/xi_meta.MYD
9554252 /var/lib/mysql/nagiosxi/xi_meta.TMD
2349356 /var/lib/mysql/nagios
1980128 /var/log
1271704 /var/log/snmptt
1189320 /var/log/snmptt/snmptt.log

[lmiltchev]

Truncating a large db table could take a while. It's going to take even longer, when you are trying to truncate 3 tables at once... Here's what you need to do:

1. Delete some old log files from /var/log. Make some room.
2. Make sure you have plenty of space in /tmp
3. Try truncating one table at the time. Truncating shouldn't hang your system but it may take a very long time. Be patient and again, make sure that you have some free space in /tmp and /var BEFORE running the truncate command.

Let us know if this helped.