Page 1 of 2
Maintenance may not be working
Posted: Mon Mar 21, 2016 10:30 am
by mike4vr
We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
Optimize indexes older than: 2 days
Close indexes older than: 30 days
Delete indexes older than: 60 days
Delete backups older than: 180 days
The backups are being done to a mounted NFS share and not on local disk.
Is there anything you can recommend checking?
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 12:24 pm
by hsmith
How long has this behavior been happening for?
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 2:19 pm
by mike4vr
hsmith wrote:How long has this behavior been happening for?
Months. I changed the "Delete indexes older than" from 90 to 60 days a few weeks ago. When the scheduler ran, it shaved off quite a bit. However, the size keeps growing. It is now larger than it was when I retained 90 days of indexes.
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 2:25 pm
by jolson
We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
Can you identify what exactly is growing? Do you know if it's your backups that are growing wildly, or is it the data that you're receiving in Nagios Log Server? Any additional information or screenshots you could provide us would be very useful in diagnosing the problem.
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 3:15 pm
by mike4vr
jolson wrote:We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
Can you identify what exactly is growing? Do you know if it's your backups that are growing wildly, or is it the data that you're receiving in Nagios Log Server? Any additional information or screenshots you could provide us would be very useful in diagnosing the problem.
The backup repository is not on local disk. It is on a mounted NFS share.
There are days where we generate more logs in record numbers, but it isn't consistent. The only consistency we have is the disk usage going up by 1% every 24 hours on 1TB disk. What sort of information and/or screenshots would you like for me to provide?
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 3:18 pm
by hsmith
This command should be helpful: find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Re: Maintenance may not be working
Posted: Mon Mar 21, 2016 3:23 pm
by mike4vr
hsmith wrote:This command should be helpful: find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Code: Select all
1.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/1/index/_osj_Lucene41_0.tim
1.3G /mnt/repo/indices/logstash-2016.03.17/4/__3r
1.3G /mnt/repo/indices/logstash-2016.03.10/4/__46
1.3G /mnt/repo/indices/logstash-2016.03.09/0/__3w
1.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/2/index/_pke_Lucene41_0.tim
1.3G /mnt/repo/indices/logstash-2016.03.08/0/__4h
1.4G /mnt/repo/indices/logstash-2016.03.17/0/__4f
1.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/4/index/_pfy_Lucene41_0.tim
1.4G /mnt/repo/indices/logstash-2016.03.10/3/__3z
1.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.10/3/index/_p5q_Lucene41_0.tim
/mnt/repo is the mounted nfs share.
Here's where my disk usage is:
Code: Select all
# du -sh --exclude=/mnt/repo --exclude=/proc /*
8.9M /bin
60M /boot
168K /dev
30M /etc
48K /home
261M /lib
26M /lib64
16K /lost+found
4.0K /media
0 /misc
4.0K /mnt
0 /net
91M /opt
556K /rhome
2.4M /root
13M /sbin
4.0K /selinux
4.0K /srv
356M /store
0 /sys
701M /tmp
587G /usr
511M /var
Here's where the usage is:
Code: Select all
# du -sh /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/*
612K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/kibana-int
8.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.22
4.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.23
3.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.24
7.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.25
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.26
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.27
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.28
8.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.29
4.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.30
3.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.31
8.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.01
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.02
9.0G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.03
8.8G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.04
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.05
4.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.06
3.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.07
7.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.08
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.09
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.10
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.11
8.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.12
4.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.13
3.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.14
7.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.15
8.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.16
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.17
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.18
8.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.19
4.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.20
3.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.21
8.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.22
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.23
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.24
9.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.25
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.26
5.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.27
4.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.28
9.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.29
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.01
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.02
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.03
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.04
6.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.05
5.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.06
13G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.07
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.08
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.10
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.11
12G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.12
10G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.13
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.14
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.15
16G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.16
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.17
19G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.18
12G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.19
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.20
16G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.21
912K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.11.25
420K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/nagioslogserver
227M /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/nagioslogserver_log
Hope that helps a bit.
Re: Maintenance may not be working
Posted: Tue Mar 22, 2016 11:29 am
by hsmith
I was looking for something growing out of control. Can you try to go to Administration --> System --> Command Subsystem and hit the Reset All Jobs button? I just want to make sure this is working correctly.
Re: Maintenance may not be working
Posted: Tue Mar 22, 2016 12:33 pm
by mike4vr
hsmith wrote:I was looking for something growing out of control. Can you try to go to Administration --> System --> Command Subsystem and hit the Reset All Jobs button? I just want to make sure this is working correctly.
Sure. Just did that. Says "Subsystem jobs have been reset." Should we wait a day now?
Re: Maintenance may not be working
Posted: Tue Mar 22, 2016 1:32 pm
by hsmith
I would wait a day to see if you see that 1% growth.