Nagios and space v2

[jsalsbury]

Good morning,
We ran into another space issue this weekend and the weekend support teams went to delete some files from a tmp folder and they ran into an issue where they would not go away to free up space.

root@XXXXXXXX:/var/tmp#
mysqld 3905 mysql 4u REG 253,5 0 98 /var/tmp/ibS2IHmu (deleted)
mysqld 3905 mysql 5u REG 253,5 0 99 /var/tmp/ib9B87jL (deleted)
mysqld 3905 mysql 6u REG 253,5 0 100 /var/tmp/ibaHNyh2 (deleted)
mysqld 3905 mysql 7u REG 253,5 0 101 /var/tmp/ibknm8eA (deleted)
mysqld 3905 mysql 11u REG 253,5 0 102 /var/tmp/ib198xmR (deleted)
mysqld 3905 mysql 20u REG 253,5 843252988 91 /var/tmp/ST7ougGj (deleted)
mysqld 3905 mysql 60u REG 253,5 843251712 94 /var/tmp/STCpUdae (deleted)

mysql27:27:SQL-TP,Service ID,AR_00000,Active,03206259,MySQL User:/var/lib/mysql:/sbin/nologin

root@XXXXXXXX:/var/tmp# ps -ef|grep -i 3905
mysql 3905 1823 5 Mar17 ? 11:15:29 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root 31763 31297 0 23:09 pts/1 00:00:00 grep --color=auto -i 3905

It looks like with Nagios growing in my company we are running into more space issues.

Firstly was there a reason support couldn't delete those files and should I look at purchasing more storage?

thanks

[dchurch]

Processes put temporary files in either /tmp or /var/tmp for use while they're running. Sometimes the program accidentally doesn't clean up its temporary files, at which point you see a temporary file being left there.

On the face of it, any file in /var/tmp or /tmp that's a) hasn't been modified in over a day, and b) isn't currently open, should be 100% fine to delete without any consequence.

Use "fuser /var/tmp/*" to check if any files are currently opened before deleting, or you can use this command:

Code: Select all

find /var/tmp/ /tmp/ -maxdepth 1 -type f -mtime +1 ! -exec fuser -s -- {} \; -exec rm -i -- {} \;

From your output, it looks like the files in /var/tmp are still open by the mysql process, even though some of them are marked as deleted. Until the file handle is closed, the space on the hard disk will not be freed. Same goes for ramdisks and really and file system. See this question for more information on how Linux frees file system space when a file is deleted.

It looks like the deleted files /var/tmp/ST7ougGj and /var/tmp/STCpUdae are two 800+MB files in an unlinked-but-not-freed state. You may be able to free up some space by restarting mysql:

Code: Select all

service mariadb restart
service mysqld restart

Many distros simply clear out the /tmp/ directory upon reboot, so rebooting is also an option.