We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
Optimize indexes older than: 2 days
Close indexes older than: 30 days
Delete indexes older than: 60 days
Delete backups older than: 180 days
The backups are being done to a mounted NFS share and not on local disk.
Is there anything you can recommend checking?
Maintenance may not be working
Re: Maintenance may not be working
How long has this behavior been happening for?
Former Nagios Employee.
me.
me.
Re: Maintenance may not be working
Months. I changed the "Delete indexes older than" from 90 to 60 days a few weeks ago. When the scheduler ran, it shaved off quite a bit. However, the size keeps growing. It is now larger than it was when I retained 90 days of indexes.hsmith wrote:How long has this behavior been happening for?
Re: Maintenance may not be working
Can you identify what exactly is growing? Do you know if it's your backups that are growing wildly, or is it the data that you're receiving in Nagios Log Server? Any additional information or screenshots you could provide us would be very useful in diagnosing the problem.We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
Re: Maintenance may not be working
The backup repository is not on local disk. It is on a mounted NFS share.jolson wrote:Can you identify what exactly is growing? Do you know if it's your backups that are growing wildly, or is it the data that you're receiving in Nagios Log Server? Any additional information or screenshots you could provide us would be very useful in diagnosing the problem.We are seeing roughly a 10GB increase every day in our storage even though we have the following Maintenance settings:
There are days where we generate more logs in record numbers, but it isn't consistent. The only consistency we have is the disk usage going up by 1% every 24 hours on 1TB disk. What sort of information and/or screenshots would you like for me to provide?
Re: Maintenance may not be working
This command should be helpful: find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Former Nagios Employee.
me.
me.
Re: Maintenance may not be working
hsmith wrote:This command should be helpful: find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Code: Select all
1.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/1/index/_osj_Lucene41_0.tim
1.3G /mnt/repo/indices/logstash-2016.03.17/4/__3r
1.3G /mnt/repo/indices/logstash-2016.03.10/4/__46
1.3G /mnt/repo/indices/logstash-2016.03.09/0/__3w
1.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/2/index/_pke_Lucene41_0.tim
1.3G /mnt/repo/indices/logstash-2016.03.08/0/__4h
1.4G /mnt/repo/indices/logstash-2016.03.17/0/__4f
1.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09/4/index/_pfy_Lucene41_0.tim
1.4G /mnt/repo/indices/logstash-2016.03.10/3/__3z
1.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.10/3/index/_p5q_Lucene41_0.timHere's where my disk usage is:
Code: Select all
# du -sh --exclude=/mnt/repo --exclude=/proc /*
8.9M /bin
60M /boot
168K /dev
30M /etc
48K /home
261M /lib
26M /lib64
16K /lost+found
4.0K /media
0 /misc
4.0K /mnt
0 /net
91M /opt
556K /rhome
2.4M /root
13M /sbin
4.0K /selinux
4.0K /srv
356M /store
0 /sys
701M /tmp
587G /usr
511M /varCode: Select all
# du -sh /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/*
612K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/kibana-int
8.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.22
4.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.23
3.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.24
7.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.25
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.26
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.27
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.28
8.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.29
4.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.30
3.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.01.31
8.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.01
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.02
9.0G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.03
8.8G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.04
8.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.05
4.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.06
3.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.07
7.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.08
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.09
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.10
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.11
8.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.12
4.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.13
3.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.14
7.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.15
8.7G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.16
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.17
9.1G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.18
8.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.19
4.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.20
3.6G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.21
8.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.22
9.2G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.23
9.3G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.24
9.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.25
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.26
5.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.27
4.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.28
9.4G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.02.29
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.01
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.02
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.03
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.04
6.5G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.05
5.9G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.06
13G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.07
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.08
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.09
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.10
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.11
12G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.12
10G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.13
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.14
18G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.15
16G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.16
20G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.17
19G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.18
12G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.19
11G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.20
16G /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.03.21
912K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/logstash-2016.11.25
420K /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/nagioslogserver
227M /usr/local/nagioslogserver/elasticsearch/data/98ec1d45-73e0-4296-ba7f-d7d71953b7e8/nodes/0/indices/nagioslogserver_logRe: Maintenance may not be working
I was looking for something growing out of control. Can you try to go to Administration --> System --> Command Subsystem and hit the Reset All Jobs button? I just want to make sure this is working correctly.
Former Nagios Employee.
me.
me.
Re: Maintenance may not be working
Sure. Just did that. Says "Subsystem jobs have been reset." Should we wait a day now?hsmith wrote:I was looking for something growing out of control. Can you try to go to Administration --> System --> Command Subsystem and hit the Reset All Jobs button? I just want to make sure this is working correctly.
Re: Maintenance may not be working
I would wait a day to see if you see that 1% growth.
Former Nagios Employee.
me.
me.