Page 1 of 1
Primary partition out of space
Posted: Fri Jun 05, 2020 12:19 pm
by jvaira
My primary partition ran out of space over night and it looks like the culprit is a single log file located in Var/log/elasticsearch. The name of the file is 6146005a-2b46-45cd-acf7-d6c4d20f245b.log and it is about 32gb. Wondering if this file is safe to delete and also what may have caused it to grow so fast to prevent this from happening again. Unfortunately the file is too large to open to see what is inside.
Re: Primary partition out of space
Posted: Fri Jun 05, 2020 3:28 pm
by ssax
What I would do is first figure out what's filling up that file:
Code: Select all
tail -n1000 /var/log/elasticsearch/6146005a-2b46-45cd-acf7-d6c4d20f245b.log
This should clean that out:
Code: Select all
service elasticsearch stop
rm -f /var/log/elasticsearch/6146005a-2b46-45cd-acf7-d6c4d20f245b.log
service elasticsearch start
Then monitor it, if it continues to grow fast, please send us a snippet of what's going into the log that is filling it up.
Once you've done that, please PM me a copy of your profile, you can download it from Admin > System Status by clicking the Download System Profile button​.
Let us know the results.
Thank you
Re: Primary partition out of space
Posted: Fri Jun 05, 2020 5:44 pm
by jvaira
Sean,
After removing the log file and restarting elastic search the machine will no longer boot up. I am assuming this is due to an issue caused by the primary partition running out of space. On boot I get the following messages over and over.
systemd-fsck[689] /dev/sdb1: recovering journal
Info: task fsck.ext4:700 blocked for more than 120 seconds
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disabled this message
Re: Primary partition out of space
Posted: Mon Jun 08, 2020 9:17 am
by ssax
If that's not working on boot you likely will need to boot into recovery mode and run an fsck.
Is this system a VM or a physical machine? What OS/version?
What specifically was done? Deleting the file and rebooting would not cause filesystem corruption unless you are uncleanly shutting it down (hard power off, etc). This is almost acting like a disk issue.
What is the output of these commands (if you're able):
Code: Select all
df -h
df -i
dmesg
uname -a
cat /etc/*release
What files are filling up the system?
Code: Select all
du -sh /usr/local/nagioslogserver/var/log/*
du -sh /var/log/*
Please send me a copy of your profile (if you're able to), you can download it from Admin > System Status by clicking the Download System Profile button​.
Create a ticket for this and include a link back to this forum thread so we can get a remote session setup:
https://support.nagios.com/tickets/
Re: Primary partition out of space
Posted: Mon Jun 08, 2020 11:05 am
by jvaira
Hello Sean,
I have created ticket 694429 and included a link back to this thread.
Re: Primary partition out of space
Posted: Mon Jun 08, 2020 11:10 am
by scottwilkerson
jvaira wrote:Hello Sean,
I have created ticket 694429 and included a link back to this thread.
Great, Locking thread, moving to ticket.