Storage Being Used Outside of Logserver

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
tvoll
Posts: 39
Joined: Fri Aug 16, 2019 9:06 am

Storage Being Used Outside of Logserver

Post by tvoll »

I have an exclusive VM for Nagios Log Server with over 2TB of storage, with the current cluster size being slightly under 500GB; however, my server is reporting that the majority of the space is being used. I'm unable to see what could be taking up so much space. Clearing snapshots or indices does not reclaim all of the space. Simple recursive commands to find largest files and directories also comes up short, not showing anything useful.
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: Storage Being Used Outside of Logserver

Post by gsmith »

Hi

Please try this

Create a file in /tmp called mydiskusage.sh that contains:

Code: Select all

#! /bin/sh

if [[ $1  == "/" ]]
then
    for dir in $(ls $1)
    do
        dupath="$1""$dir"
        if [[ $dupath != "/proc" ]]
        then
            du -sh $dupath
        fi
    done
else
    for dir in $(ls $1)
    do
        dupath="$1"/"$dir"
        du -sh $dupath
    done
fi

Then you should be able to drill down based on it's output like:
[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /
0 /bin
242M /boot
0 /dev
27M /etc
12K /home
0 /lib
0 /lib64
0 /media
0 /mnt
0 /opt
76M /root
17M /run
0 /sbin
0 /srv
2.1M /store
0 /sys
24K /tmp
3.3G /usr
708M /var
[root@gs-cent8-23-82 tmp]#



So now I'd go to /usr

[root@gs-cent8-23-82 tmp]# /tmp/mydiskusage.sh /usr
251M /usr/bin
0 /usr/games
39M /usr/include
1.1G /usr/lib
474M /usr/lib64
191M /usr/libexec
684M /usr/local
46M /usr/sbin
589M /usr/share
0 /usr/src
0 /usr/tmp
[root@gs-cent8-23-82 tmp]#



It's pretty crude put it should allow you to find out where the space is being used.

Thanks
tvoll
Posts: 39
Joined: Fri Aug 16, 2019 9:06 am

Re: Storage Being Used Outside of Logserver

Post by tvoll »

It definitely helped! It appears that the indices weren't properly rotating even with my maintenance settings set to 14 days for my chosen repository. We've had it set to 14 days for maintenance and I discovered that there was a backlog of over a month. The System Jobs menu shows that all jobs have been running. Any idea to remove those old logstash indices folders?
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: Storage Being Used Outside of Logserver

Post by gsmith »

Hi

What you probably need to do is make a change to the setting "Delete indexes older than" :
Screenshot 2022-02-17 161418.png
As you can see in the tooltip, a setting of zero actually disables the deletion of the indices

Please let me know if that is what you were looking for.

Thanks
You do not have the required permissions to view the files attached to this post.
tvoll
Posts: 39
Joined: Fri Aug 16, 2019 9:06 am

Re: Storage Being Used Outside of Logserver

Post by tvoll »

Attached is what our settings have been for months now.
You do not have the required permissions to view the files attached to this post.
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: Storage Being Used Outside of Logserver

Post by gsmith »

Hi

What are the permissions on the files in the indices directory of your repo?

Here's what I have:

Code: Select all

[root@gs-cent79-23-89 indices]# pwd
/repos/repo1/indices
[root@gs-cent79-23-89 indices]# ll
total 136
drwxr-xr-x 7 nagios users  129 Jan 14 16:31 logstash-2021.12.16
drwxr-xr-x 7 nagios users  168 Jan 15 16:31 logstash-2021.12.17
drwxr-xr-x 7 nagios users  207 Jan 16 16:31 logstash-2021.12.18
drwxr-xr-x 7 nagios users  246 Jan 17 10:04 logstash-2021.12.19
drwxr-xr-x 7 nagios users  285 Jan 18 10:04 logstash-2021.12.20
drwxr-xr-x 7 nagios users  324 Jan 19 10:04 logstash-2021.12.21
drwxr-xr-x 7 nagios users  324 Jan 19 10:04 logstash-2021.12.22
drwxr-xr-x 7 nagios users  324 Jan 19 10:04 logstash-2021.12.23
drwxr-xr-x 7 nagios users  324 Jan 19 10:04 logstash-2021.12.24
Thanks
tvoll
Posts: 39
Joined: Fri Aug 16, 2019 9:06 am

Re: Storage Being Used Outside of Logserver

Post by tvoll »

Here is what I am seeing:

Code: Select all

[tvollscw@logserver indices]$ pwd
/mnt/logs-backup/indices
[tvollscw@logserver indices]$ ll
total 128
-rw-r--r-- 1 root   root   133 Feb 16 16:43 logstash-2021.12.31.tar.gz
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.27
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.28
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.29
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.30
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.01.31
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.01
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.02
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.03
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.04
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.05
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.06
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.07
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.08
drwxr-xr-x 2 nagios users 4096 Feb 24 12:36 logstash-2022.02.09
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.10
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.11
drwxr-xr-x 7 nagios users 4096 Feb 27 10:56 logstash-2022.02.12
drwxr-xr-x 7 nagios users 4096 Feb 26 10:47 logstash-2022.02.13
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.14
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.15
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.16
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.17
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.18
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.19
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.20
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.21
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.22
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.23
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.24
drwxr-xr-x 7 nagios users 4096 Feb 27 10:41 logstash-2022.02.25
drwxr-xr-x 7 nagios users 4096 Feb 27 10:53 logstash-2022.02.26
As you may notice, it isn't removing after 14 days. The indices are still present.
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: Storage Being Used Outside of Logserver

Post by gsmith »

Hi

You should delete the files that are older than 14 days. Restart the system by running:

Code: Select all

systemctl restart logstash
systemctl restart elasticsearch
systemctl restart httpd
Then see if the system only keeps files that are less than or equal to 14 days
old. If the system fails to delete a file older than 14 days please let me know.

Thanks!
tvoll
Posts: 39
Joined: Fri Aug 16, 2019 9:06 am

Re: Storage Being Used Outside of Logserver

Post by tvoll »

That is what I have been doing manually for weeks now. They just continually come back up and have an issue deleting themselves.
gsmith
Posts: 1253
Joined: Tue Mar 02, 2021 11:15 am

Re: Storage Being Used Outside of Logserver

Post by gsmith »

Hi

I'll set up a system and try to duplicate the issue.

I'll let you know what I see.

Thanks!
Locked