Ran Out Of Disk Space on centos-root

[jjordan18]

I went to go login to the GUI and was not able to get a login page. I logged into the CLI and noticed that the volume for /dev/mapper/centos-root is full (see below). What is the best way to free up space on this volume within the CLI? I do not need to worry about keeping any log history.
Then, I need to figure out how to increase the size of this partition....


Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/centos-root 52403200 52403180 20 100% /
devtmpfs 5020788 0 5020788 0% /dev
tmpfs 5034452 0 5034452 0% /dev/shm
tmpfs 5034452 549560 4484892 11% /run
tmpfs 5034452 0 5034452 0% /sys/fs/cgroup
/dev/sda1 1038336 147968 890368 15% /boot
/dev/mapper/centos-home 525923428 33036 525890392 1% /home
tmpfs 1006892 0 1006892 0% /run/user/1000
tmpfs 1006892 0 1006892 0% /run/user/1001

[npolovenko]

Hi, @jjordan18. You can start off by running this command:

find /dev/mapper/centos-root -type f -print0 | xargs -0 du | sort -n | tail -20 | cut -f2 | xargs -I{} du -sh {}

This will show top 10 space consuming files on that partition. You can post the output of this command and I'll let you know if these files are ok to delete.

Once you free up space you should consider increasing the partition size. We have an article showing how to do that on a VM based XI:
https://support.nagios.com/kb/article.php?id=266

[jjordan18]

This is the output for the command you provided:

[root@mpls01-nagios01 ~]# find /dev/mapper/centos-root -type f -print0 | xargs -0 du | sort -n | tail -20 | cut -f2 | xargs -I{} du -sh {}
48K ./.cache/pip/http/f
56K ./.cache/pip/http/2/5
56K ./.cache/pip/http/2/5/6
56K ./.cache/pip/http/2/5/6/b
56K ./.cache/pip/http/2/5/6/b/c
64K ./.cache/pip/http/2
84K ./.cache/pip/http/8/5
84K ./.cache/pip/http/8/5/7
84K ./.cache/pip/http/8/5/7/7
84K ./.cache/pip/http/8/5/7/7/a
108K ./.cache/pip/http/8
116K ./.cache/pip/http/e
116K ./.cache/pip/http/e/e
116K ./.cache/pip/http/e/e/4
116K ./.cache/pip/http/e/e/4/5
116K ./.cache/pip/http/e/e/4/5/0
428K ./.cache/pip/http
432K ./.cache
432K ./.cache/pip
464K .


Also, this is a physical server so am assuming I will need to re-spin to increase the size of the root partition? There is plenty of disk space available but the "auto install" function of CentOS made it this size. I should have sized it manually. :/

I did not get notified of the reply by you to this incident.

[npolovenko]

@jjordan18, You could try resizing logical volume sizes with LVM but it may be easier to just spin up another system. Did you have any advanced configurations on this Nagios server such as mod gearman, ram disk or offloaded database? If not it should be fairly easy to do a backup/restore to the new server:
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

[jjordan18]

No fancy config...I think it may be best to just re-spin the thing.

Thanks for your help with this.

[jjordan18]

Wait a minute....I just noticed that the "home" partition is 501.80GB. Can I just move the syslog data to this partition?

[jjordan18]

[root@mpls01-nagios01 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 centos lvm2 a-- <556.75g 4.00m
[root@mpls01-nagios01 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
centos 1 3 0 wz--n- <556.75g 4.00m
[root@mpls01-nagios01 ~]# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
home centos -wi-ao---- 501.80g
root centos -wi-ao---- 50.00g
swap centos -wi-ao---- <4.94g

[npolovenko]

@jjordan18, You could change the elasticsearch datapath and move all elasticsearch data over to the home partition. Here's the KB article:
https://assets.nagios.com/downloads/nag ... Server.pdf
Elasticsearch database takes up most space so moving it should be enough to fix the space issue.