XI Constantly Running Out of Space

[blariv]

We keep running out of space on the root volume. not sure why anymore, given this is a small volume but I used to be able to clear up space by deleting logs. now the logs are offloaded to an nfs share, backups are also doing the same. the only way I can clean out a small amount of space is by running the database repair script. i have searched the system and there are only 2 files larger than 100MB.

/var/lib/mysql/nagios/nagios_externalcommands.MYD
/var/lib/mysql/nagios/nagios_commenthistory.MYD

My database performance settings are attached.

I am running an old version of XI as well. 2012R2.7

thank you

[BanditBBS]

Check /store folder. It is where nagios stores all the backups. I create a new drive and mount there just for that reason.

[blariv]

thank you but I have already offloaded that

[BanditBBS]

Do you know what folder is getting all the data written to it that is filling your drive?

[blariv]

no i am thinking its empty space on the mysql tables. I have no idea other than that

[BanditBBS]

no i am thinking its empty space on the mysql tables. I have no idea other than that

Go to the commandline and enter:

Code: Select all

alias duh='du -sk * | sort -n | perl -ne '\''($s,$f)=split(m{\t});for (qw(K M G)) {if($s<1024) {printf("%.1f",$s);print "$_\t$f"; last};$s=$s/1024}'\'

then just type in "duh" and it'll display the list of your folders and space they are using. Then go into the folder using the most and type "duh" again, so on and so on and trace it down. I'm going to guess it is performance data.

[rkennedy]

How much disk space do you have on the machine? df -H

You may simply need to increase the partition if it's rather small. As your amount of objects grow, so do the logs with time.

This command may help to track down what's taking all the space, could you show us the output? du -a / | sort -n -r | head -n 10

[blariv]

I am not able to extend the volume are we are on physical disks. here is the output of the command, like I said earlier there arent any gigantic files.

0.0K C!MAXIPDIAL.txt
4.0K 1386302401.tar.gz
4.0K 1386907201.tar.gz
4.0K 1387684801.tar.gz
4.0K 1388548801.tar.gz
4.0K anaconda-ks.cfg
4.0K C1MAXIPDIAL.txt
4.0K C1MAXIP.TXT
4.0K hats.cfg
8.0K install.log.syslog
8.0K isilon.txt
8.0K isus
28.0K install.log
52.0K scripts
80.0K group.txt
292.0K mod_gearman-1.4.2-1.e.rhel6.x86_64.rpm
616.0K xi-latest.tar.gz
1.0M services.cfg
2.3M monitoring-plugins-2.2.tar.gz
23.1M monitoring-plugins-2.2
155.6M rpmbuild

[dwhitfield]

https://assets.nagios.com/downloads/nag ... ucture.pdf may help you in your search.

yum clean packages may also help. yum clean all is more drastic, but may be necessary.

I know some of the logs are offloaded, but what's the output of ll /var/log/?

[blariv]

still nothing major.

total 23076
-rw-------. 1 root root 34575 Nov 5 2012 anaconda.ifcfg.log
-rw-------. 1 root root 19868 Nov 5 2012 anaconda.log
-rw-------. 1 root root 70452 Nov 5 2012 anaconda.program.log
-rw-------. 1 root root 135861 Nov 5 2012 anaconda.storage.log
-rw-------. 1 root root 129121 Nov 5 2012 anaconda.syslog
-rw-------. 1 root root 44803 Nov 5 2012 anaconda.xlog
-rw-------. 1 root root 60415 Nov 5 2012 anaconda.yum.log
drwxr-x---. 2 root root 4096 Jun 30 2016 audit
-rw-r--r-- 1 root root 3646 Jun 5 2016 boot.log
-rw------- 1 root utmp 18816 Jan 31 04:42 btmp
-rw------- 1 root utmp 9216 Dec 24 00:13 btmp-20170101
drwxr-xr-x. 2 root root 4096 Nov 5 2012 ConsoleKit
drwxr-xr-x. 2 lp sys 4096 Jan 28 03:42 cups
drwxr-xr-x 4 root root 4096 Jul 12 2014 dell
-rw-r--r-- 1 root root 70912 Jun 5 2016 dmesg
-rw-r--r-- 1 root root 73023 Aug 25 2015 dmesg.old
-rw-r--r-- 1 root root 184887 May 16 2016 dracut.log
drwxr-xr-x 2 gearmand gearmand 4096 Jan 10 2013 gearmand
drwx------. 2 root root 4096 Nov 23 03:46 httpd
-rw-r--r--. 1 root root 147752 Jan 31 08:59 lastlog
-rw------- 1 root root 18520513 Jan 31 12:50 maillog
-rw-r--r--. 1 root root 0 Nov 5 2012 mcelog
drwxr-xr-x 2 nagios root 4096 Jun 5 2016 mod_gearman
-rw-r-----. 1 mysql mysql 2850768 Jan 31 12:52 mysqld.log
drwxr-xr-x. 2 ntp ntp 4096 Jan 25 2016 ntpstats
drwxr-x--- 2 pe-puppet pe-puppet 4096 Oct 28 2015 pe-puppet
drwxr-xr-x. 2 root root 4096 Aug 19 2013 prelink
drwxr-xr-x 4 root root 4096 Jan 29 03:11 puppetlabs
drwxr-xr-x. 2 root root 4096 Jan 31 00:00 sa
drwx------. 3 root root 4096 Apr 12 2016 samba
-rw------- 1 root root 509851 Jan 31 11:28 secure
-rw------- 1 root root 13671 May 15 2016 secure-20160515
-rw------- 1 root root 16782 Jun 19 2016 secure-20160619
-rw------- 1 root root 22430 Jun 26 2016 secure-20160626
-rw-r--r-- 1 root root 169226 Jun 29 2014 snmpd.log
drwxrwxr-x 2 snmptt snmptt 4096 Jan 29 03:11 snmptt
-rw------- 1 root root 0 Jun 26 2016 spooler
-rw------- 1 root root 0 May 8 2016 spooler-20160515
-rw------- 1 root root 0 May 15 2016 spooler-20160612
-rw------- 1 root root 0 Jun 12 2016 spooler-20160619
-rw------- 1 root root 0 Jun 19 2016 spooler-20160626
drwxr-x---. 2 root root 4096 Mar 22 2016 sssd
-rw-------. 1 root root 0 Nov 5 2012 tallylog
-rw-rw-r--. 1 root utmp 425472 Jan 31 08:59 wtmp
-rw------- 1 root root 281 Jan 12 14:58 yum.log
-rw------- 1 root root 8925 Oct 13 2013 yum.log-20140101
-rw------- 1 root root 411 Feb 20 2014 yum.log-20150101
-rw------- 1 root root 5279 Dec 17 2015 yum.log-20160101
-rw------- 1 root root 33458 Dec 13 14:08 yum.log-20170101