Page 1 of 2
Storage Being Used Outside of Logserver
Posted: Thu Jan 27, 2022 11:58 am
by tvoll
I have an exclusive VM for Nagios Log Server with over 2TB of storage, with the current cluster size being slightly under 500GB; however, my server is reporting that the majority of the space is being used. I'm unable to see what could be taking up so much space. Clearing snapshots or indices does not reclaim all of the space. Simple recursive commands to find largest files and directories also comes up short, not showing anything useful.
Re: Storage Being Used Outside of Logserver
Posted: Fri Jan 28, 2022 12:21 pm
by gsmith
Hi
Please try this
Create a file in /tmp called mydiskusage.sh that contains:
Code: Select all
#! /bin/sh
if [[ $1 == "/" ]]
then
for dir in $(ls $1)
do
dupath="$1""$dir"
if [[ $dupath != "/proc" ]]
then
du -sh $dupath
fi
done
else
for dir in $(ls $1)
do
dupath="$1"/"$dir"
du -sh $dupath
done
fi
Then you should be able to drill down based on it's output like:
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /
0 /bin
242M /boot
0 /dev
27M /etc
12K /home
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
76M /root
17M /run
0 /sbin
0 /srv
2.1M /store
0 /sys
24K /tmp
3.3G /usr
708M /var
[root@gs-cent8-23-82 tmp]#
So now I'd go to /usr
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /usr
251M /usr/bin
0 /usr/games
39M /usr/include
1.1G /usr/lib
474M /usr/lib64
191M /usr/libexec
684M /usr/local
46M /usr/sbin
589M /usr/share
0 /usr/src
0 /usr/tmp
[root@gs-cent8-23-82 tmp]#
It's pretty crude put it should allow you to find out where the space is being used.
Thanks
Re: Storage Being Used Outside of Logserver
Posted: Wed Feb 16, 2022 5:52 pm
by tvoll
It definitely helped! It appears that the indices weren't properly rotating even with my maintenance settings set to 14 days for my chosen repository. We've had it set to 14 days for maintenance and I discovered that there was a backlog of over a month. The System Jobs menu shows that all jobs have been running. Any idea to remove those old logstash indices folders?
Re: Storage Being Used Outside of Logserver
Posted: Thu Feb 17, 2022 5:19 pm
by gsmith
Hi
What you probably need to do is make a change to the setting "Delete indexes older than" :
Screenshot 2022-02-17 161418.png
As you can see in the tooltip, a setting of zero actually disables the deletion of the indices
Please let me know if that is what you were looking for.
Thanks
Re: Storage Being Used Outside of Logserver
Posted: Wed Feb 23, 2022 9:59 am
by tvoll
Attached is what our settings have been for months now.
Re: Storage Being Used Outside of Logserver
Posted: Wed Feb 23, 2022 1:22 pm
by gsmith
Hi
What are the permissions on the files in the indices directory of your repo?
Here's what I have:
Code: Select all
[root@gs-cent79-23-89 indices]# pwd
/repos/repo1/indices
[root@gs-cent79-23-89 indices]# ll
total 136
drwxr-xr-x 7 nagios users 129 Jan 14 16:31 logstash-2021.12.16
drwxr-xr-x 7 nagios users 168 Jan 15 16:31 logstash-2021.12.17
drwxr-xr-x 7 nagios users 207 Jan 16 16:31 logstash-2021.12.18
drwxr-xr-x 7 nagios users 246 Jan 17 10:04 logstash-2021.12.19
drwxr-xr-x 7 nagios users 285 Jan 18 10:04 logstash-2021.12.20
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.21
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.22
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.23
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.24
Thanks
Re: Storage Being Used Outside of Logserver
Posted: Mon Feb 28, 2022 9:31 am
by tvoll
Here is what I am seeing:
Code: Select all
[tvollscw@logserver indices]$ pwd
/mnt/logs-backup/indices
[tvollscw@logserver indices]$ ll
total 128
-rw-r--r-- 1 root root 133 Feb 16 16:43 logstash-2021.12.31.tar.gz
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.27
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.28
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.29
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.30
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.31
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.01
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.02
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.03
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.04
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.05
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.06
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.07
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.08
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.09
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.10
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.11
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.12
drwxr-xr-x 7 nagios users 4096 Feb 26 10:47 logstash-2022.02.13
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.14
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.15
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.16
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.17
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.18
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.19
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.20
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.21
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.22
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.23
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.24
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.25
drwxr-xr-x 7 nagios users 4096 Feb 27 10:53 logstash-2022.02.26
As you may notice, it isn't removing after 14 days. The indices are still present.
Re: Storage Being Used Outside of Logserver
Posted: Wed Mar 02, 2022 3:44 pm
by gsmith
Hi
You should delete the files that are older than 14 days. Restart the system by running:
Code: Select all
systemctl restart logstash
systemctl restart elasticsearch
systemctl restart httpd
Then see if the system only keeps files that are less than or equal to 14 days
old. If the system fails to delete a file older than 14 days please let me know.
Thanks!
Re: Storage Being Used Outside of Logserver
Posted: Thu Mar 03, 2022 9:54 am
by tvoll
That is what I have been doing manually for weeks now. They just continually come back up and have an issue deleting themselves.
Re: Storage Being Used Outside of Logserver
Posted: Thu Mar 03, 2022 1:57 pm
by gsmith
Hi
I'll set up a system and try to duplicate the issue.
I'll let you know what I see.
Thanks!