100GB disk full after one week?

[TDBruno]

Hello,

New to this forum and Nagios products.
I started running the Trial version of log server last Monday to check if we could use this in our environment. It is installed with 100GB available space.
Initially, it was only collecting logs from itself and one other PC.
On Friday, I noticed the elasticsearch stopped working and rebooted the server to get it back up.
That same day, I added 3 firewalls for it to log.

Today, I log into the server and see that again the elasticsearch stopped working AND all 100GB have been filled by the logs.

I see under Index Status, the largest logs are from Friday and Saturday, 349.9MB and 304.5 MB respectively.

How could it be that 100GB worth of logs have been collected in a week from so few hosts? Is there a way to filter what logs it keeps?

[hsmith]

Can you post the output of anls -l /var/log/ command?

[TDBruno]

Here is the output.

[hsmith]

Here's a horrible command for you to run:

Code: Select all

find / -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}

[TDBruno]

Ok, here you go. Think I got that all right.

[hsmith]

What's the output of a df -h command?

[TDBruno]

Here it is...and it seems I misread the actual size of the logs before.

So it's not close to 100GB but is larger than I expected after one week.
I'm curious about the log file sizes after adding the firewalls.
Today's log file has already grown to 380MB.
Does that seem right?

[hsmith]

How much information is your firewall spewing out? I just got out of a remote support session with a customer who is receiving roughly 19GB of logs per day.

[TDBruno]

I know there is a lot of information produced by those firewalls but am not sure how much. I will look into that and post back here.
While trying to log on the server again, I received the Elasticsearch error message.
Is this related?

This screenshot shows the output on the console.

logs4.jpg

This screenshot shows the output after running service elasticsearch start

logs5.jpg

[rkennedy]

Looks like you're running out of memory. What is the result of these two commands?

Code: Select all

free -m
top|head -5