Page 1 of 2

100GB disk full after one week?

Posted: Mon Feb 08, 2016 9:30 am
by TDBruno
Hello,

New to this forum and Nagios products.
I started running the Trial version of log server last Monday to check if we could use this in our environment. It is installed with 100GB available space.
Initially, it was only collecting logs from itself and one other PC.
On Friday, I noticed the elasticsearch stopped working and rebooted the server to get it back up.
That same day, I added 3 firewalls for it to log.

Today, I log into the server and see that again the elasticsearch stopped working AND all 100GB have been filled by the logs.

I see under Index Status, the largest logs are from Friday and Saturday, 349.9MB and 304.5 MB respectively.

How could it be that 100GB worth of logs have been collected in a week from so few hosts? Is there a way to filter what logs it keeps?

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 10:42 am
by hsmith
Can you post the output of anls -l /var/log/ command?

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 11:23 am
by TDBruno
Here is the output.

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 11:56 am
by hsmith
Here's a horrible command for you to run:

Code: Select all

find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 12:10 pm
by TDBruno
Ok, here you go. Think I got that all right.

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 12:17 pm
by hsmith
What's the output of a df -h command?

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 12:41 pm
by TDBruno
Here it is...and it seems I misread the actual size of the logs before. :oops:

So it's not close to 100GB but is larger than I expected after one week.
I'm curious about the log file sizes after adding the firewalls.
Today's log file has already grown to 380MB.
Does that seem right?

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 12:43 pm
by hsmith
How much information is your firewall spewing out? I just got out of a remote support session with a customer who is receiving roughly 19GB of logs per day.

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 2:54 pm
by TDBruno
I know there is a lot of information produced by those firewalls but am not sure how much. I will look into that and post back here.
While trying to log on the server again, I received the Elasticsearch error message.
Is this related?

This screenshot shows the output on the console.
logs4.jpg
This screenshot shows the output after running service elasticsearch start
logs5.jpg

Re: 100GB disk full after one week?

Posted: Mon Feb 08, 2016 3:00 pm
by rkennedy
Looks like you're running out of memory. What is the result of these two commands?

Code: Select all

free -m
top|head -5