Primary partition out of space

[jvaira]

My primary partition ran out of space over night and it looks like the culprit is a single log file located in Var/log/elasticsearch. The name of the file is 6146005a-2b46-45cd-acf7-d6c4d20f245b.log and it is about 32gb. Wondering if this file is safe to delete and also what may have caused it to grow so fast to prevent this from happening again. Unfortunately the file is too large to open to see what is inside.

[ssax]

What I would do is first figure out what's filling up that file:

Code: Select all

tail -n1000 /var/log/elasticsearch/6146005a-2b46-45cd-acf7-d6c4d20f245b.log

This should clean that out:

Code: Select all

service elasticsearch stop
rm -f /var/log/elasticsearch/6146005a-2b46-45cd-acf7-d6c4d20f245b.log
service elasticsearch start

Then monitor it, if it continues to grow fast, please send us a snippet of what's going into the log that is filling it up.

Once you've done that, please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button​.

Let us know the results.

Thank you

[jvaira]

Sean,
After removing the log file and restarting elastic search the machine will no longer boot up. I am assuming this is due to an issue caused by the primary partition running out of space. On boot I get the following messages over and over.
systemd-fsck[689] /dev/sdb1: recovering journal
Info: task fsck.ext4:700 blocked for more than 120 seconds
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disabled this message

[ssax]

If that's not working on boot you likely will need to boot into recovery mode and run an fsck.

Is this system a VM or a physical machine? What OS/version?

What specifically was done? Deleting the file and rebooting would not cause filesystem corruption unless you are uncleanly shutting it down (hard power off, etc). This is almost acting like a disk issue.

What is the output of these commands (if you're able):

Code: Select all

df -h
df -i
dmesg
uname -a
cat /etc/*release

What files are filling up the system?

Code: Select all

du -sh /usr/local/nagioslogserver/var/log/*
du -sh /var/log/*

Please send me a copy of your profile (if you're able to), you can download it from Admin > System Status by clicking the Download System Profile button​.

Create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/

[jvaira]

Hello Sean,
I have created ticket 694429 and included a link back to this thread.

[scottwilkerson]

Hello Sean,
I have created ticket 694429 and included a link back to this thread.

Great, Locking thread, moving to ticket.