Storage Being Used Outside of Logserver
Storage Being Used Outside of Logserver
I have an exclusive VM for Nagios Log Server with over 2TB of storage, with the current cluster size being slightly under 500GB; however, my server is reporting that the majority of the space is being used. I'm unable to see what could be taking up so much space. Clearing snapshots or indices does not reclaim all of the space. Simple recursive commands to find largest files and directories also comes up short, not showing anything useful.
Re: Storage Being Used Outside of Logserver
Hi
Please try this
Create a file in /tmp called mydiskusage.sh that contains:
Then you should be able to drill down based on it's output like:
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /
0 /bin
242M /boot
0 /dev
27M /etc
12K /home
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
76M /root
17M /run
0 /sbin
0 /srv
2.1M /store
0 /sys
24K /tmp
3.3G /usr
708M /var
[root@gs-cent8-23-82 tmp]#
So now I'd go to /usr
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /usr
251M /usr/bin
0 /usr/games
39M /usr/include
1.1G /usr/lib
474M /usr/lib64
191M /usr/libexec
684M /usr/local
46M /usr/sbin
589M /usr/share
0 /usr/src
0 /usr/tmp
[root@gs-cent8-23-82 tmp]#
It's pretty crude put it should allow you to find out where the space is being used.
Thanks
Please try this
Create a file in /tmp called mydiskusage.sh that contains:
Code: Select all
#! /bin/sh
if [[ $1 == "/" ]]
then
for dir in $(ls $1)
do
dupath="$1""$dir"
if [[ $dupath != "/proc" ]]
then
du -sh $dupath
fi
done
else
for dir in $(ls $1)
do
dupath="$1"/"$dir"
du -sh $dupath
done
fi
Then you should be able to drill down based on it's output like:
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /
0 /bin
242M /boot
0 /dev
27M /etc
12K /home
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
76M /root
17M /run
0 /sbin
0 /srv
2.1M /store
0 /sys
24K /tmp
3.3G /usr
708M /var
[root@gs-cent8-23-82 tmp]#
So now I'd go to /usr
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /usr
251M /usr/bin
0 /usr/games
39M /usr/include
1.1G /usr/lib
474M /usr/lib64
191M /usr/libexec
684M /usr/local
46M /usr/sbin
589M /usr/share
0 /usr/src
0 /usr/tmp
[root@gs-cent8-23-82 tmp]#
It's pretty crude put it should allow you to find out where the space is being used.
Thanks
Re: Storage Being Used Outside of Logserver
It definitely helped! It appears that the indices weren't properly rotating even with my maintenance settings set to 14 days for my chosen repository. We've had it set to 14 days for maintenance and I discovered that there was a backlog of over a month. The System Jobs menu shows that all jobs have been running. Any idea to remove those old logstash indices folders?
Re: Storage Being Used Outside of Logserver
Hi
What you probably need to do is make a change to the setting "Delete indexes older than" :
As you can see in the tooltip, a setting of zero actually disables the deletion of the indices
Please let me know if that is what you were looking for.
Thanks
What you probably need to do is make a change to the setting "Delete indexes older than" :
As you can see in the tooltip, a setting of zero actually disables the deletion of the indices
Please let me know if that is what you were looking for.
Thanks
You do not have the required permissions to view the files attached to this post.
Re: Storage Being Used Outside of Logserver
Attached is what our settings have been for months now.
You do not have the required permissions to view the files attached to this post.
Re: Storage Being Used Outside of Logserver
Hi
What are the permissions on the files in the indices directory of your repo?
Here's what I have:
Thanks
What are the permissions on the files in the indices directory of your repo?
Here's what I have:
Code: Select all
[root@gs-cent79-23-89 indices]# pwd
/repos/repo1/indices
[root@gs-cent79-23-89 indices]# ll
total 136
drwxr-xr-x 7 nagios users 129 Jan 14 16:31 logstash-2021.12.16
drwxr-xr-x 7 nagios users 168 Jan 15 16:31 logstash-2021.12.17
drwxr-xr-x 7 nagios users 207 Jan 16 16:31 logstash-2021.12.18
drwxr-xr-x 7 nagios users 246 Jan 17 10:04 logstash-2021.12.19
drwxr-xr-x 7 nagios users 285 Jan 18 10:04 logstash-2021.12.20
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.21
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.22
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.23
drwxr-xr-x 7 nagios users 324 Jan 19 10:04 logstash-2021.12.24
Re: Storage Being Used Outside of Logserver
Here is what I am seeing:
As you may notice, it isn't removing after 14 days. The indices are still present.
Code: Select all
[tvollscw@logserver indices]$ pwd
/mnt/logs-backup/indices
[tvollscw@logserver indices]$ ll
total 128
-rw-r--r-- 1 root root 133 Feb 16 16:43 logstash-2021.12.31.tar.gz
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.27
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.28
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.29
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.30
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.31
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.01
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.02
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.03
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.04
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.05
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.06
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.07
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.08
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.09
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.10
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.11
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.12
drwxr-xr-x 7 nagios users 4096 Feb 26 10:47 logstash-2022.02.13
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.14
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.15
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.16
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.17
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.18
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.19
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.20
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.21
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.22
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.23
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.24
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.25
drwxr-xr-x 7 nagios users 4096 Feb 27 10:53 logstash-2022.02.26
Re: Storage Being Used Outside of Logserver
Hi
You should delete the files that are older than 14 days. Restart the system by running:
Then see if the system only keeps files that are less than or equal to 14 days
old. If the system fails to delete a file older than 14 days please let me know.
Thanks!
You should delete the files that are older than 14 days. Restart the system by running:
Code: Select all
systemctl restart logstash
systemctl restart elasticsearch
systemctl restart httpd
old. If the system fails to delete a file older than 14 days please let me know.
Thanks!
Re: Storage Being Used Outside of Logserver
That is what I have been doing manually for weeks now. They just continually come back up and have an issue deleting themselves.
Re: Storage Being Used Outside of Logserver
Hi
I'll set up a system and try to duplicate the issue.
I'll let you know what I see.
Thanks!
I'll set up a system and try to duplicate the issue.
I'll let you know what I see.
Thanks!