Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

Disk space is filling up repidly

https://support.nagios.com/forum/viewtopic.php?t=56636

Page 2 of 2

Re: Disk space is filling up repidly

Posted: Wed Dec 04, 2019 11:44 am
by snapon_admin

Code: Select all

[root@lisl-ngos-01-pv nagiosxi]# find / -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
du: cannot access `/proc/8282/task/8282/fd/4': No such file or directory
du: cannot access `/proc/8282/task/8282/fdinfo/4': No such file or directory
du: cannot access `/proc/8282/fd/4': No such file or directory
du: cannot access `/proc/8282/fdinfo/4': No such file or directory
du: cannot access `/proc/10973': No such file or directory
du: cannot access `/proc/10983': No such file or directory
du: cannot access `/proc/10985': No such file or directory
du: cannot access `/proc/10988': No such file or directory
du: cannot access `/proc/10992': No such file or directory
du: cannot access `/proc/10993': No such file or directory
du: cannot access `/proc/10995': No such file or directory
du: cannot access `/proc/10996': No such file or directory
du: cannot access `/proc/10997': No such file or directory
du: cannot access `/proc/11000': No such file or directory
du: cannot access `/proc/11006': No such file or directory
du: cannot access `/proc/11007': No such file or directory
du: cannot access `/proc/11008': No such file or directory
du: cannot access `/proc/11009': No such file or directory
du: cannot access `/proc/11010': No such file or directory
du: cannot access `/proc/11011': No such file or directory
find: `/proc/9797': No such file or directory
find: `/proc/9797': No such file or directory
find: `/proc/11522': No such file or directory
find: `/proc/11522': No such file or directory
find: `/proc/11885': No such file or directory
find: `/proc/11885': No such file or directory
find: `/proc/11961': No such file or directory
find: `/proc/11961': No such file or directory
find: `/proc/11962': No such file or directory
find: `/proc/11962': No such file or directory
find: `/proc/11965': No such file or directory
find: `/proc/11965': No such file or directory
find: `/proc/11974': No such file or directory
find: `/proc/11974': No such file or directory
find: `/proc/11975': No such file or directory
find: `/proc/11975': No such file or directory
find: `/proc/11976': No such file or directory
find: `/proc/11976': No such file or directory
find: `/proc/11979': No such file or directory
find: `/proc/11979': No such file or directory
find: `/proc/11980': No such file or directory
find: `/proc/11980': No such file or directory
find: `/proc/11981': No such file or directory
find: `/proc/11981': No such file or directory
du: cannot access `/proc/8136': No such file or directory
du: cannot access `/proc/9838': No such file or directory
du: cannot access `/proc/11996': No such file or directory
du: cannot access `/proc/12001': No such file or directory
du: cannot access `/proc/12002': No such file or directory
du: cannot access `/proc/12004': No such file or directory
du: cannot access `/proc/12007': No such file or directory
du: cannot access `/proc/12008': No such file or directory
du: cannot access `/proc/12012': No such file or directory
du: cannot access `/proc/12013': No such file or directory
du: cannot access `/proc/12017': No such file or directory
du: cannot access `/proc/12018': No such file or directory
du: cannot access `/proc/12019': No such file or directory
du: cannot access `/proc/12020': No such file or directory
du: cannot access `/proc/12021': No such file or directory
du: cannot access `/proc/12022/task/12022/fd/4': No such file or directory
du: cannot access `/proc/12022/task/12022/fdinfo/4': No such file or directory
du: cannot access `/proc/12022/fd/4': No such file or directory
du: cannot access `/proc/12022/fdinfo/4': No such file or directory
du: cannot access `/proc/12023': No such file or directory
du: cannot access `/proc/8136': No such file or directory
du: cannot access `/proc/8136/task': No such file or directory
du: cannot access `/proc/8136/task/8136': No such file or directory
du: cannot access `/proc/8136/task/8136/fd': No such file or directory
du: cannot access `/proc/8136/task/8136/fdinfo': No such file or directory
du: cannot access `/proc/8136/task/8136/ns': No such file or directory
du: cannot access `/proc/8136/task/8136/attr': No such file or directory
du: cannot access `/proc/8136/fd': No such file or directory
du: cannot access `/proc/8136/fdinfo': No such file or directory
du: cannot access `/proc/8136/ns': No such file or directory
du: cannot access `/proc/8136/net': No such file or directory
du: cannot access `/proc/8136/net/dev_snmp6': No such file or directory
du: cannot access `/proc/8136/net/netfilter': No such file or directory
du: cannot access `/proc/8136/net/stat': No such file or directory
du: cannot access `/proc/8136/attr': No such file or directory
du: cannot access `/proc/9838': No such file or directory
du: cannot access `/proc/9838/task': No such file or directory
du: cannot access `/proc/9838/task/9838': No such file or directory
du: cannot access `/proc/9838/task/9838/fd': No such file or directory
du: cannot access `/proc/9838/task/9838/fdinfo': No such file or directory
du: cannot access `/proc/9838/task/9838/ns': No such file or directory
du: cannot access `/proc/9838/task/9838/attr': No such file or directory
du: cannot access `/proc/9838/fd': No such file or directory
du: cannot access `/proc/9838/fdinfo': No such file or directory
du: cannot access `/proc/9838/ns': No such file or directory
du: cannot access `/proc/9838/net': No such file or directory
du: cannot access `/proc/9838/net/dev_snmp6': No such file or directory
du: cannot access `/proc/9838/net/netfilter': No such file or directory
du: cannot access `/proc/9838/net/stat': No such file or directory
du: cannot access `/proc/9838/attr': No such file or directory
du: cannot access `/proc/9839': No such file or directory
du: cannot access `/proc/9839/task': No such file or directory
du: cannot access `/proc/9839/task/9839': No such file or directory
du: cannot access `/proc/9839/task/9839/fd': No such file or directory
du: cannot access `/proc/9839/task/9839/fdinfo': No such file or directory
du: cannot access `/proc/9839/task/9839/ns': No such file or directory
du: cannot access `/proc/9839/task/9839/attr': No such file or directory
du: cannot access `/proc/9839/fd': No such file or directory
du: cannot access `/proc/9839/fdinfo': No such file or directory
du: cannot access `/proc/9839/ns': No such file or directory
du: cannot access `/proc/9839/net': No such file or directory
du: cannot access `/proc/9839/net/dev_snmp6': No such file or directory
du: cannot access `/proc/9839/net/netfilter': No such file or directory
du: cannot access `/proc/9839/net/stat': No such file or directory
du: cannot access `/proc/9839/attr': No such file or directory
du: cannot access `/proc/9911': No such file or directory
du: cannot access `/proc/9911/task': No such file or directory
du: cannot access `/proc/9911/task/9911': No such file or directory
du: cannot access `/proc/9911/task/9911/fd': No such file or directory
du: cannot access `/proc/9911/task/9911/fdinfo': No such file or directory
du: cannot access `/proc/9911/task/9911/ns': No such file or directory
du: cannot access `/proc/9911/task/9911/attr': No such file or directory
du: cannot access `/proc/9911/fd': No such file or directory
du: cannot access `/proc/9911/fdinfo': No such file or directory
du: cannot access `/proc/9911/ns': No such file or directory
du: cannot access `/proc/9911/net': No such file or directory
du: cannot access `/proc/9911/net/dev_snmp6': No such file or directory
du: cannot access `/proc/9911/net/netfilter': No such file or directory
du: cannot access `/proc/9911/net/stat': No such file or directory
du: cannot access `/proc/9911/attr': No such file or directory
du: cannot access `/proc/11393': No such file or directory
du: cannot access `/proc/11393/task': No such file or directory
du: cannot access `/proc/11393/task/11393': No such file or directory
du: cannot access `/proc/11393/task/11393/fd': No such file or directory
du: cannot access `/proc/11393/task/11393/fdinfo': No such file or directory
du: cannot access `/proc/11393/task/11393/ns': No such file or directory
du: cannot access `/proc/11393/task/11393/attr': No such file or directory
du: cannot access `/proc/11393/fd': No such file or directory
du: cannot access `/proc/11393/fdinfo': No such file or directory
du: cannot access `/proc/11393/ns': No such file or directory
du: cannot access `/proc/11393/net': No such file or directory
du: cannot access `/proc/11393/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11393/net/netfilter': No such file or directory
du: cannot access `/proc/11393/net/stat': No such file or directory
du: cannot access `/proc/11393/attr': No such file or directory
du: cannot access `/proc/11394': No such file or directory
du: cannot access `/proc/11394/task': No such file or directory
du: cannot access `/proc/11394/task/11394': No such file or directory
du: cannot access `/proc/11394/task/11394/fd': No such file or directory
du: cannot access `/proc/11394/task/11394/fdinfo': No such file or directory
du: cannot access `/proc/11394/task/11394/ns': No such file or directory
du: cannot access `/proc/11394/task/11394/attr': No such file or directory
du: cannot access `/proc/11394/fd': No such file or directory
du: cannot access `/proc/11394/fdinfo': No such file or directory
du: cannot access `/proc/11394/ns': No such file or directory
du: cannot access `/proc/11394/net': No such file or directory
du: cannot access `/proc/11394/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11394/net/netfilter': No such file or directory
du: cannot access `/proc/11394/net/stat': No such file or directory
du: cannot access `/proc/11394/attr': No such file or directory
du: cannot access `/proc/11598': No such file or directory
du: cannot access `/proc/11598/task': No such file or directory
du: cannot access `/proc/11598/task/11598': No such file or directory
du: cannot access `/proc/11598/task/11598/fd': No such file or directory
du: cannot access `/proc/11598/task/11598/fdinfo': No such file or directory
du: cannot access `/proc/11598/task/11598/ns': No such file or directory
du: cannot access `/proc/11598/task/11598/attr': No such file or directory
du: cannot access `/proc/11598/fd': No such file or directory
du: cannot access `/proc/11598/fdinfo': No such file or directory
du: cannot access `/proc/11598/ns': No such file or directory
du: cannot access `/proc/11598/net': No such file or directory
du: cannot access `/proc/11598/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11598/net/netfilter': No such file or directory
du: cannot access `/proc/11598/net/stat': No such file or directory
du: cannot access `/proc/11598/attr': No such file or directory
du: cannot access `/proc/11682': No such file or directory
du: cannot access `/proc/11682/task': No such file or directory
du: cannot access `/proc/11682/task/11682': No such file or directory
du: cannot access `/proc/11682/task/11682/fd': No such file or directory
du: cannot access `/proc/11682/task/11682/fdinfo': No such file or directory
du: cannot access `/proc/11682/task/11682/ns': No such file or directory
du: cannot access `/proc/11682/task/11682/attr': No such file or directory
du: cannot access `/proc/11682/fd': No such file or directory
du: cannot access `/proc/11682/fdinfo': No such file or directory
du: cannot access `/proc/11682/ns': No such file or directory
du: cannot access `/proc/11682/net': No such file or directory
du: cannot access `/proc/11682/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11682/net/netfilter': No such file or directory
du: cannot access `/proc/11682/net/stat': No such file or directory
du: cannot access `/proc/11682/attr': No such file or directory
du: cannot access `/proc/11690': No such file or directory
du: cannot access `/proc/11690/task': No such file or directory
du: cannot access `/proc/11690/task/11690': No such file or directory
du: cannot access `/proc/11690/task/11690/fd': No such file or directory
du: cannot access `/proc/11690/task/11690/fdinfo': No such file or directory
du: cannot access `/proc/11690/task/11690/ns': No such file or directory
du: cannot access `/proc/11690/task/11690/attr': No such file or directory
du: cannot access `/proc/11690/fd': No such file or directory
du: cannot access `/proc/11690/fdinfo': No such file or directory
du: cannot access `/proc/11690/ns': No such file or directory
du: cannot access `/proc/11690/net': No such file or directory
du: cannot access `/proc/11690/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11690/net/netfilter': No such file or directory
du: cannot access `/proc/11690/net/stat': No such file or directory
du: cannot access `/proc/11690/attr': No such file or directory
du: cannot access `/proc/11950': No such file or directory
du: cannot access `/proc/11950/task': No such file or directory
du: cannot access `/proc/11950/task/11950': No such file or directory
du: cannot access `/proc/11950/task/11950/fd': No such file or directory
du: cannot access `/proc/11950/task/11950/fdinfo': No such file or directory
du: cannot access `/proc/11950/task/11950/ns': No such file or directory
du: cannot access `/proc/11950/task/11950/attr': No such file or directory
du: cannot access `/proc/11950/fd': No such file or directory
du: cannot access `/proc/11950/fdinfo': No such file or directory
du: cannot access `/proc/11950/ns': No such file or directory
du: cannot access `/proc/11950/net': No such file or directory
du: cannot access `/proc/11950/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11950/net/netfilter': No such file or directory
du: cannot access `/proc/11950/net/stat': No such file or directory
du: cannot access `/proc/11950/attr': No such file or directory
du: cannot access `/proc/11952': No such file or directory
du: cannot access `/proc/11952/task': No such file or directory
du: cannot access `/proc/11952/task/11952': No such file or directory
du: cannot access `/proc/11952/task/11952/fd': No such file or directory
du: cannot access `/proc/11952/task/11952/fdinfo': No such file or directory
du: cannot access `/proc/11952/task/11952/ns': No such file or directory
du: cannot access `/proc/11952/task/11952/attr': No such file or directory
du: cannot access `/proc/11952/fd': No such file or directory
du: cannot access `/proc/11952/fdinfo': No such file or directory
du: cannot access `/proc/11952/ns': No such file or directory
du: cannot access `/proc/11952/net': No such file or directory
du: cannot access `/proc/11952/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11952/net/netfilter': No such file or directory
du: cannot access `/proc/11952/net/stat': No such file or directory
du: cannot access `/proc/11952/attr': No such file or directory
du: cannot access `/proc/11973': No such file or directory
du: cannot access `/proc/11973/task': No such file or directory
du: cannot access `/proc/11973/task/11973': No such file or directory
du: cannot access `/proc/11973/task/11973/fd': No such file or directory
du: cannot access `/proc/11973/task/11973/fdinfo': No such file or directory
du: cannot access `/proc/11973/task/11973/ns': No such file or directory
du: cannot access `/proc/11973/task/11973/attr': No such file or directory
du: cannot access `/proc/11973/fd': No such file or directory
du: cannot access `/proc/11973/fdinfo': No such file or directory
du: cannot access `/proc/11973/ns': No such file or directory
du: cannot access `/proc/11973/net': No such file or directory
du: cannot access `/proc/11973/net/dev_snmp6': No such file or directory
du: cannot access `/proc/11973/net/netfilter': No such file or directory
du: cannot access `/proc/11973/net/stat': No such file or directory
du: cannot access `/proc/11973/attr': No such file or directory
du: cannot access `/var/spool/snmptt/#snmptt-trap-1575477824151351': No such file or directory
59G     /usr/local/nagios
59G     /usr/local/nagios
70G     /usr/local
70G     /usr/local
70G     /usr/local
72G     /usr
72G     /usr
du: cannot access `/var/spool/snmptt/#snmptt-trap-1575477828895350': No such file or directory
84G     /var
du: cannot access `/var/spool/snmptt/#snmptt-trap-1575477829271062': No such file or directory
84G     /var
du: cannot access `/proc/12223': No such file or directory
du: cannot access `/proc/12224': No such file or directory
du: cannot access `/proc/12861': No such file or directory
du: cannot access `/proc/12869': No such file or directory
du: cannot access `/proc/12907/task/12907/fd/4': No such file or directory
du: cannot access `/proc/12907/task/12907/fdinfo/4': No such file or directory
du: cannot access `/proc/12907/fd/4': No such file or directory
du: cannot access `/proc/12907/fdinfo/4': No such file or directory
du: cannot access `/proc/12909': No such file or directory
du: cannot access `/proc/12911': No such file or directory
du: cannot access `/proc/12941': No such file or directory
du: cannot access `/proc/12943': No such file or directory
du: cannot access `/proc/12946': No such file or directory
du: cannot access `/proc/12952': No such file or directory
du: cannot access `/proc/12957': No such file or directory
du: cannot access `/proc/12959': No such file or directory
du: cannot access `/proc/12966': No such file or directory
du: cannot access `/proc/12967': No such file or directory
du: cannot access `/proc/12968': No such file or directory
197G    /
[root@lisl-ngos-01-pv nagiosxi]# find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find: `/proc/13666': No such file or directory
find: `/proc/13666': No such file or directory
find: `/proc/13671': No such file or directory
find: `/proc/13671': No such file or directory
find: `/proc/13708': No such file or directory
find: `/proc/13708': No such file or directory
find: `/proc/13732': No such file or directory
find: `/proc/13732': No such file or directory
find: `/proc/13760': No such file or directory
find: `/proc/13760': No such file or directory
find: `/proc/13762': No such file or directory
find: `/proc/13762': No such file or directory
find: `/proc/13766': No such file or directory
find: `/proc/13766': No such file or directory
find: `/proc/13767': No such file or directory
find: `/proc/13767': No such file or directory
find: `/proc/13774': No such file or directory
find: `/proc/13774': No such file or directory
find: `/proc/13775': No such file or directory
find: `/proc/13775': No such file or directory
find: `/proc/13781': No such file or directory
find: `/proc/13781': No such file or directory
find: `/proc/13782': No such file or directory
find: `/proc/13782': No such file or directory
find: `/proc/13783': No such file or directory
find: `/proc/13783': No such file or directory
find: `/proc/13786': No such file or directory
find: `/proc/13786': No such file or directory
find: `/proc/13788': No such file or directory
find: `/proc/13788': No such file or directory
find: `/proc/13789': No such file or directory
find: `/proc/13789': No such file or directory
find: `/proc/13790': No such file or directory
find: `/proc/13790': No such file or directory
du: cannot access `/proc/13436/task/13436/fdinfo/5': No such file or directory
du: cannot access `/proc/13436/fdinfo/5': No such file or directory
du: cannot access `/proc/17718/task/17718/fdinfo/12': No such file or directory
du: cannot access `/proc/17718/task/17718/fdinfo/16': No such file or directory
du: cannot access `/proc/17718/fdinfo/12': No such file or directory
du: cannot access `/proc/17718/fdinfo/16': No such file or directory
du: cannot access `/proc/17720/task/17720/fdinfo/5': No such file or directory
du: cannot access `/proc/17720/task/17720/fdinfo/11': No such file or directory
du: cannot access `/proc/17720/fdinfo/5': No such file or directory
du: cannot access `/proc/17720/fdinfo/11': No such file or directory
du: cannot access `/proc/17721/task/17721/fdinfo/8': No such file or directory
du: cannot access `/proc/17721/task/17721/fdinfo/12': No such file or directory
du: cannot access `/proc/17721/fdinfo/8': No such file or directory
du: cannot access `/proc/17721/fdinfo/12': No such file or directory
du: cannot access `/proc/17724/task/17724/fdinfo/7': No such file or directory
du: cannot access `/proc/17724/task/17724/fdinfo/12': No such file or directory
du: cannot access `/proc/17724/fdinfo/7': No such file or directory
du: cannot access `/proc/17724/fdinfo/12': No such file or directory
3.0G    /var/log/snmptt/snmptthandler.debug
3.1G    /var/log/snmptt/snmptt.debug-20191202
4.3G    /var/lib/mysql/nagios/nagios_logentries.MYI
6.4G    /var/log/snmptt/snmptthandler.debug-20191201
6.6G    /store/backups/nagiosxi/nagiosxi.1575252002.tar.gz
6.7G    /var/lib/mysql/nagios/nagios_logentries.MYD
6.7G    /store/backups/nagiosxi/nagiosxi.1575338401.tar.gz
6.8G    /store/backups/nagiosxi/nagiosxi.1575424802.tar.gz
19G     /var/spool/mail/root
25G     /var/log/snmptt/snmptt.debug

Re: Disk space is filling up repidly

Posted: Wed Dec 04, 2019 3:12 pm
by tgriep
These are the 2 largest files on the server.

Code: Select all

19G     /var/spool/mail/root
25G     /var/log/snmptt/snmptt.debug

The /var/spool/mail/root is the root's email inbox so it you are reading them, truncate the file.

The other is the snmptt.debug file.
If you are not looking at that, stop it from logging in the /etc/snmp/snmptt.ini file and remove the log file.

While you are at it, disable the snmptthandler.debug settings for the snmptt daemon and remove these files.

Code: Select all

3.0G    /var/log/snmptt/snmptthandler.debug
6.4G    /var/log/snmptt/snmptthandler.debug-20191201
That should free up some space.

The commands don's show all of the subfolders so you will have to adjust it to see further down the folder tree.

Code: Select all

find /usr/local/nagios -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
find /var -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Instead of starting at the root / partition, the above examples start at /usr/local/nagios and /var.

Re: Disk space is filling up repidly

Posted: Wed Dec 04, 2019 4:16 pm
by snapon_admin
I did disable the debugging and removed those files. Whatever is causing this, though, I don't think it has to do with that. Nor does finding the largest files seem to be helping. The disk is just constantly filling up. Removing these large files helps because it drops the total usage down, but it's still climbing even after doing that. See the attached screenshot of the graph from the day this all started until now. I included a week before so you can see this wasn't always doing this.
root usage.png

Re: Disk space is filling up repidly

Posted: Wed Dec 04, 2019 5:01 pm
by tgriep
To continue on, you would have to re-run the commands once or twice a day and save the output for comparison.
Whatever file or folder shows a continual growth after a few days, then look there and start again.

A few folders to look at are as follows.
/var/log and all if the sub folders
/var/spool and all if the sub folders
/tmp
/var/tmp
/usr/local/nagios/var
/usr/local/nagiosxi/var
Run this to see if the MYSQL tables are large.

Code: Select all

echo "SELECT table_schema as 'Database', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC;" |mysql -t -u root -pnagiosxi

Re: Disk space is filling up repidly

Posted: Thu Dec 05, 2019 11:36 am
by snapon_admin
I'm pretty convinced this is some sort of a problem with snmptt. Even though it appears to be functioning something is going on with that spool directory that is not normal. I just killed and restarted snmptt yet again and it freed up 35G. That should not be the case...

Code: Select all

[root@lisl-ngos-01-pv nagiosxi]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      295G  184G   97G  66% /
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1              97M   37M   56M  40% /boot
tmpfs                 500M   36M  465M   8% /var/nagiosramdisk
You have new mail in /var/spool/mail/root
[root@lisl-ngos-01-pv nagiosxi]# service snmptt stop
Stopping snmptt:                                           [FAILED]
[root@lisl-ngos-01-pv nagiosxi]# service snmptt start
Starting snmptt:                                           [  OK  ]
[root@lisl-ngos-01-pv nagiosxi]# ll /var/spool/snmptt
total 32
-rw-r--r--. 1 root root 728 Dec  5 10:32 #snmptt-trap-1575563572757871
-rw-r--r--. 1 root root 728 Dec  5 10:32 #snmptt-trap-1575563573084806
-rw-r--r--. 1 root root 755 Dec  5 10:32 #snmptt-trap-1575563573147127
-rw-r--r--. 1 root root 728 Dec  5 10:32 #snmptt-trap-1575563573206900
-rw-r--r--. 1 root root 758 Dec  5 10:32 #snmptt-trap-1575563573269000
-rw-r--r--. 1 root root 728 Dec  5 10:32 #snmptt-trap-1575563573331076
-rw-r--r--. 1 root root 730 Dec  5 10:32 #snmptt-trap-1575563573389521
-rw-r--r--. 1 root root 761 Dec  5 10:32 #snmptt-trap-1575563573448217
[root@lisl-ngos-01-pv nagiosxi]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      295G  149G  131G  54% /
tmpfs                  16G     0   16G   0% /dev/shm
/dev/sda1              97M   37M   56M  40% /boot
tmpfs                 500M   36M  465M   8% /var/nagiosramdisk
I'm not sure how big the DB tables should be but this doesn't seem too crazy to me:

Code: Select all

[root@lisl-ngos-01-pv nagiosxi]# echo "SELECT table_schema as 'Database', table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES ORDER BY (data_length + index_length) DESC;" |mysql -t -u root -pnagiosxi
+--------------------+--------------------------------------------+------------+
| Database           | Table                                      | Size in MB |
+--------------------+--------------------------------------------+------------+
| nagios             | nagios_logentries                          |   12040.91 |
| nagios             | nagios_externalcommands                    |    2295.71 |
| nagios             | nagios_commenthistory                      |      52.63 |
| nagios             | nagios_statehistory                        |      45.06 |
| nagios             | nagios_downtimehistory                     |      11.46 |
| nagios             | nagios_servicestatus                       |       6.72 |
| nagios             | nagios_objects                             |       6.44 |
| nagios             | nagios_contactnotifications                |       5.39 |
| nagios             | nagios_contactnotificationmethods          |       5.11 |
| nagios             | nagios_services                            |       2.66 |
| nagiosql           | tbl_service                                |       2.55 |
| nagios             | nagios_flappinghistory                     |       1.96 |
| nagios             | nagios_service_contactgroups               |       1.79 |
| nagios             | nagios_notifications                       |       1.58 |
| nagiosql           | tbl_variabledefinition                     |       1.34 |
| nagios             | nagios_conninfo                            |       1.24 |
| nagios             | nagios_customvariablestatus                |       0.92 |
| nagios             | nagios_customvariables                     |       0.87 |
| nagios             | nagios_hoststatus                          |       0.66 |
| nagios             | nagios_acknowledgements                    |       0.60 |
| mysql              | help_topic                                 |       0.43 |
| nagiosql           | tbl_lnkServiceToServicetemplate            |       0.43 |
| nagiosql           | tbl_lnkServiceToHost                       |       0.36 |
| nagios             | nagios_processevents                       |       0.35 |
| nagiosql           | tbl_lnkServiceToVariabledefinition         |       0.32 |
| nagios             | nagios_hosts                               |       0.31 |
| nagiosql           | tbl_host                                   |       0.28 |
| nagios             | nagios_hostgroup_members                   |       0.19 |
| nagios             | nagios_host_contactgroups                  |       0.16 |
| nagios             | nagios_contact_notificationcommands        |       0.14 |
| nagiosql           | tbl_info                                   |       0.13 |
| nagiosql           | tbl_lnkHostToContactgroup                  |       0.10 |
| nagios             | nagios_timeperiod_timeranges               |       0.10 |
| mysql              | help_keyword                               |       0.10 |
| nagiosql           | tbl_timedefinition                         |       0.10 |
| nagiosql           | tbl_lnkServiceToContactgroup               |       0.10 |
| nagios             | nagios_comments                            |       0.07 |
| nagiosql           | tbl_lnkHostToHostgroup                     |       0.06 |
| nagios             | nagios_service_contacts                    |       0.06 |
| nagiosql           | tbl_command                                |       0.06 |
| nagios             | nagios_systemcommands                      |       0.06 |
| nagiosql           | tbl_timeperiod                             |       0.04 |
| nagiosql           | tbl_lnkHostToHosttemplate                  |       0.04 |
| nagiosql           | tbl_contact                                |       0.04 |
| nagios             | nagios_host_parenthosts                    |       0.04 |
| nagiosql           | tbl_lnkHostgroupToHost                     |       0.04 |
| nagios             | nagios_commands                            |       0.03 |
| nagios             | nagios_contacts                            |       0.03 |
| nagios             | nagios_contactgroup_members                |       0.03 |
| nagiosql           | tbl_lnkHostToHost                          |       0.03 |
| nagiosql           | tbl_lnkHostToVariabledefinition            |       0.03 |
| nagios             | nagios_eventhandlers                       |       0.03 |
| mysql              | help_relation                              |       0.02 |
| mysql              | help_category                              |       0.02 |
| nagios             | nagios_timeperiods                         |       0.02 |
| nagios             | nagios_contactstatus                       |       0.02 |
| nagiosql           | tbl_contactgroup                           |       0.02 |
| nagiosql           | tbl_servicetemplate                        |       0.02 |
| nagiosql           | tbl_hosttemplate                           |       0.02 |
| nagiosql           | tbl_hostgroup                              |       0.01 |
| nagiosql           | tbl_lnkContactToContacttemplate            |       0.01 |
| nagios             | nagios_contactgroups                       |       0.01 |
| nagiosql           | tbl_lnkContactgroupToContact               |       0.01 |
| nagios             | nagios_configfilevariables                 |       0.01 |
| nagios             | nagios_host_contacts                       |       0.01 |
| nagiosql           | tbl_lnkServiceToContact                    |       0.01 |
| nagiosql           | tbl_user                                   |       0.01 |
| nagiosql           | tbl_servicegroup                           |       0.01 |
| nagiosql           | tbl_domain                                 |       0.01 |
| nagiosql           | tbl_contacttemplate                        |       0.01 |
| nagios             | nagios_hostgroups                          |       0.01 |
| nagiosql           | tbl_lnkContactToVariabledefinition         |       0.01 |
| mysql              | db                                         |       0.01 |
| nagiosql           | tbl_lnkHostToContact                       |       0.01 |
| mysql              | proc                                       |       0.01 |
| nagiosql           | tbl_lnkServicetemplateToServicetemplate    |       0.01 |
| nagiosql           | tbl_lnkHostgroupToHostgroup                |       0.00 |
| nagios             | nagios_configfiles                         |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHosttemplate          |       0.00 |
| nagiosql           | tbl_serviceescalation                      |       0.00 |
| mysql              | columns_priv                               |       0.00 |
| nagiosql           | tbl_servicedependency                      |       0.00 |
| mysql              | tables_priv                                |       0.00 |
| nagiosql           | tbl_hostescalation                         |       0.00 |
| mysql              | procs_priv                                 |       0.00 |
| nagiosql           | tbl_hostdependency                         |       0.00 |
| nagios             | nagios_scheduleddowntime                   |       0.00 |
| nagios             | nagios_runtimevariables                    |       0.00 |
| nagiosql           | tbl_submenu                                |       0.00 |
| nagiosql           | tbl_session                                |       0.00 |
| nagios             | nagios_servicegroup_members                |       0.00 |
| nagiosql           | tbl_settings                               |       0.00 |
| nagios             | nagios_servicegroups                       |       0.00 |
| nagios             | nagios_programstatus                       |       0.00 |
| nagios             | nagios_contact_addresses                   |       0.00 |
| nagiosql           | tbl_lnkServicegroupToService               |       0.00 |
| mysql              | user                                       |       0.00 |
| nagiosql           | tbl_session_locks                          |       0.00 |
| nagiosql           | tbl_mainmenu                               |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContactgroup          |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContactgroup       |       0.00 |
| nagiosql           | tbl_lnkServiceToServicegroup               |       0.00 |
| nagiosql           | tbl_lnkContactToCommandHost                |       0.00 |
| nagiosql           | tbl_lnkContactgroupToContactgroup          |       0.00 |
| nagiosql           | tbl_lnkContactToContactgroup               |       0.00 |
| nagiosql           | tbl_lnkContactToCommandService             |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandService     |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToCommandHost        |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToVariabledefinition    |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHostgroup             |       0.00 |
| nagios             | nagios_instances                           |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToVariabledefinition |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContactgroup       |       0.00 |
| mysql              | host                                       |       0.00 |
| mysql              | event                                      |       0.00 |
| nagios             | nagios_dbversion                           |       0.00 |
| information_schema | PROCESSLIST                                |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToContact               |       0.00 |
| mysql              | ndb_binlog_index                           |       0.00 |
| nagiosql           | tbl_serviceextinfo                         |       0.00 |
| nagios             | nagios_hostdependencies                    |       0.00 |
| mysql              | time_zone_leap_second                      |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_H      |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToServicegroup       |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToService          |       0.00 |
| nagios             | nagios_hostchecks                          |       0.00 |
| mysql              | time_zone                                  |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHostgroup_DH     |       0.00 |
| nagios             | nagios_serviceescalations                  |       0.00 |
| information_schema | PLUGINS                                    |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHostgroup          |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToHostgroup        |       0.00 |
| nagios             | nagios_timedeventqueue                     |       0.00 |
| information_schema | COLUMNS                                    |       0.00 |
| information_schema | VIEWS                                      |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_H           |       0.00 |
| nagios             | nagios_serviceescalation_contacts          |       0.00 |
| information_schema | PARTITIONS                                 |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToHost             |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToHost_DH          |       0.00 |
| nagios             | nagios_serviceescalation_contactgroups     |       0.00 |
| nagiosql           | tbl_lnkHostescalationToHostgroup           |       0.00 |
| nagiosql           | tbl_hostextinfo                            |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_DH        |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToContactgroup     |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToHost               |       0.00 |
| mysql              | servers                                    |       0.00 |
| information_schema | TRIGGERS                                   |       0.00 |
| nagios             | nagios_servicedependencies                 |       0.00 |
| nagiosql           | tbl_lnkHostescalationToHost                |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToVariabledefinition |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_H              |       0.00 |
| mysql              | time_zone_transition_type                  |       0.00 |
| nagiosql           | tbl_lnkServiceescalationToContact          |       0.00 |
| information_schema | ROUTINES                                   |       0.00 |
| nagiosql           | tbl_logbook                                |       0.00 |
| nagios             | nagios_servicechecks                       |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContactgroup        |       0.00 |
| nagios             | nagios_hostescalations                     |       0.00 |
| nagiosql           | tbl_lnkContacttemplateToContacttemplate    |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHost_DH             |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_S        |       0.00 |
| nagiosql           | tbl_lnkHosttemplateToHost                  |       0.00 |
| nagiosql           | tbl_lnkTimeperiodToTimeperiod              |       0.00 |
| nagiosql           | tbl_lnkServicetemplateToContact            |       0.00 |
| nagios             | nagios_service_parentservices              |       0.00 |
| nagiosql           | tbl_lnkHostescalationToContact             |       0.00 |
| nagios             | nagios_hostescalation_contacts             |       0.00 |
| mysql              | time_zone_transition                       |       0.00 |
| mysql              | func                                       |       0.00 |
| nagios             | nagios_timedevents                         |       0.00 |
| nagiosql           | tbl_lnkServicegroupToServicegroup          |       0.00 |
| mysql              | plugin                                     |       0.00 |
| nagiosql           | tbl_lnkServiceToHostgroup                  |       0.00 |
| information_schema | EVENTS                                     |       0.00 |
| nagiosql           | tbl_lnkHostdependencyToHostgroup_H         |       0.00 |
| nagios             | nagios_hostescalation_contactgroups        |       0.00 |
| mysql              | time_zone_name                             |       0.00 |
| nagiosql           | tbl_lnkServicedependencyToService_DS       |       0.00 |
| information_schema | STATISTICS                                 |       0.00 |
| information_schema | ENGINES                                    |       0.00 |
| information_schema | SESSION_VARIABLES                          |       0.00 |
| information_schema | COLUMN_PRIVILEGES                          |       0.00 |
| information_schema | SESSION_STATUS                             |       0.00 |
| information_schema | SCHEMA_PRIVILEGES                          |       0.00 |
| information_schema | COLLATION_CHARACTER_SET_APPLICABILITY      |       0.00 |
| mysql              | slow_log                                   |       0.00 |
| information_schema | USER_PRIVILEGES                            |       0.00 |
| information_schema | KEY_COLUMN_USAGE                           |       0.00 |
| information_schema | SCHEMATA                                   |       0.00 |
| information_schema | COLLATIONS                                 |       0.00 |
| information_schema | GLOBAL_VARIABLES                           |       0.00 |
| mysql              | general_log                                |       0.00 |
| information_schema | CHARACTER_SETS                             |       0.00 |
| information_schema | TABLE_PRIVILEGES                           |       0.00 |
| information_schema | GLOBAL_STATUS                              |       0.00 |
| information_schema | REFERENTIAL_CONSTRAINTS                    |       0.00 |
| information_schema | TABLE_CONSTRAINTS                          |       0.00 |
| information_schema | FILES                                      |       0.00 |
| information_schema | PROFILING                                  |       0.00 |
| information_schema | TABLES                                     |       0.00 |
+--------------------+--------------------------------------------+------------+

Re: Disk space is filling up repidly

Posted: Thu Dec 05, 2019 12:39 pm
by tgriep
When you tried to stop snmptt it failed as it was not running so if snmptt was not running, the Trap spool files will build up in the /var/spool/snmptt/ folder.
Then when it was started, it processed them and removed them which is why the space was restored.

Look in the log files in the /var/log/snmptt folder to see when and why the snmptt daemon was not running.

Re: Disk space is filling up repidly

Posted: Thu Dec 05, 2019 1:12 pm
by snapon_admin
That's the issue, though. I didn't check the status this time but the last time I did this I ran 'service snmptt status' and it said it was running. I also checked the spool directory and files were being generated and appeared to be automatically removed (I'd check and there'd be ~10 files, then check again and it'd be say 30, then check again and it'd be back down to 15) but when I run 'service snmptt stop' it says failed even though the status command just told me it was running.

Re: Disk space is filling up repidly

Posted: Thu Dec 05, 2019 1:59 pm
by tgriep
It could be how snmptt was killed that cause the stop to fail when it was ran.

Next time, run the process command to verify that snmptt is running or not.

Code: Select all

ps -ef |grep snmptt
Then if you see a buildup of the Trap files in the spool folder, check the log files in the /var/log/snmptt folder for any errors.
Also, before restarting or killing anything, run this and post it here.

Code: Select all

ps -ef |grep snmp
All times are UTC-05:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited