Page 1 of 2
Root Partition is getting full every other day
Posted: Wed Feb 23, 2022 6:00 am
by G_Deshpande
Hello Team
On Two different Nagios servers We are monitoring 1500 hosts and 10k services in RHEL 7 Linux box.
Drive size of /dev/mapper/rhel-root is 95 GB on both Nagios server and is getting full on one of the Nagios server.
We did tried to remove old backups and stuff like that however getting same issue on frequent basis.
Can you help us which particular logs are safe to remove like from path /var/log/ or from /usr/local/nagios/ etc?
Because we don't want to loose any important data. please specify path and file names to be removed.
Also help us in how much more disk space is required to get added. means how much additional VM space we need for this environment to avoid such space crunch issue in future as we keep on adding more host and services day by day.
Regards
Gajanan
Re: Root Partition is getting full every other day
Posted: Wed Feb 23, 2022 6:29 pm
by kfanselow
Hi Gajanan,
What version of XI are you currently using ? Let's start by gathering some more information about where you are using the space. Could you run the following on your XI server and attach the output file (/tmp/diskusage.txt ) to this thread ?
Code: Select all
sudo du -sh /usr/local/nagios /usr/local/nagiosxi /usr/local/nagvis > /tmp/diskusage.txt
sudo du -sh /usr/local/nagiosxi/nom /usr/local/nagios/share/perfdata >> /tmp/diskusage.txt
printf "\n\n ====================================================== \n\n" >> /tmp/diskusage.txt
sudo du -sh /var/log /var/run/ /var/spool /var/lib/mysql /var/lib/mrtg/ /store/backups/mysql /store/backups/nagiosxi >> /tmp/diskusage.txt
printf "\n\n ====================================================== \n\n" >> /tmp/diskusage.txt
find /usr/local/nagiosxi/var -type f -exec du -sh {} \; >> /tmp/diskusage.txt
printf "\n\n ====================================================== \n\n" >> /tmp/diskusage.txt
find /store/ -type d -exec du -sh {} \; >> /tmp/diskusage.txt
printf "\n\n ====================================================== \n\n" >> /tmp/diskusage.txt
echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -uroot -pnagiosxi --table >> /tmp/diskusage.txt
Thanks and Best Regards,
Keith
Re: Root Partition is getting full every other day
Posted: Tue Mar 01, 2022 10:46 am
by G_Deshpande
Thanks for the revert.
We are using 5.8.7 version of Nagios XI and i will run those command and provide you the output.
Re: Root Partition is getting full every other day
Posted: Tue Mar 01, 2022 11:05 am
by G_Deshpande
Hey
I have executed mentioned command and found that below directories/folder consuming more disk-
49G /usr/local/nagios
15G /usr/local/nagiosxi
2.9G /var/log
Output is attached here in thread.
Thanks!
Re: Root Partition is getting full every other day
Posted: Wed Mar 02, 2022 5:57 pm
by kfanselow
Hi Gajanan,
That paints a much more clear picture of where the data is being used. The output in /var/log is probably being managed by logrotate ( check with your OS distribution for specifics ) and you may want to review the settings, your organization's retention policy, and what your operational need is for those files.
Let's drill down a little bit more in the nagios directories; I suspect we'll see log archives and performance data as the main culprits. It would also be good to take a look at /store as well. Please run the following and attach the output file ( /tmp/du-nagios.txt ) to the thread:
Code: Select all
sudo uname -a
sudo find /usr/local/nagios/ -type d -exec du -sh {} \; | grep "M\|G" > /tmp/du-nagios.txt
sudo printf "\n\n ====================================================== \n\n" >> /tmp/du-nagios.txt
sudo find /usr/local/nagiosxi/ -type d -exec du -sh {} \; | grep "M\|G" >> /tmp/du-nagios.txt
sudo printf "\n\n ====================================================== \n\n" >> /tmp/du-nagios.txt
sudo du -sh /store/ >> /tmp/du-nagios.txt
Thanks and Best Regards,
Keith
Re: Root Partition is getting full every other day
Posted: Thu Mar 03, 2022 6:38 am
by G_Deshpande
Hey Keith
Find attached output.
Issue was started on 18th or 19th Feb before that we didn't have root partition alert. also find attached state history report of root partition.
This will help to narrow-down the issue that issue started on 18th so that we can check logs of same day.
We have another Nagios box where we don't have this issue on it.
Regards
Gajanan
Re: Root Partition is getting full every other day
Posted: Thu Mar 03, 2022 8:10 am
by G_Deshpande
Hi
/usr/local/nagios/var/archives
consumes 45 GB of space. find attached logs for same directory.
Regards
Gajanan
Re: Root Partition is getting full every other day
Posted: Fri Mar 04, 2022 4:51 pm
by kfanselow
Hi Gajanan,
Thank you for sending that on. It looks like there is a marked change in the size of the files starting on the 23rd of February. I think this is something we should look into further. Please file a support ticket to help us better assist you with this issue. Opening a ticket allows us to better prioritize your problem and communicate more confidentially with you about your configuration and the problems you are encountering. You can reference this forum thread in the ticket, however a simple summary of the problem, and a fresh system profile, also help to bring support team members up to speed on your issue.
Here is a link to our ticketing system:
https://support.nagios.com/tickets/
Thank you for your patience and understanding -- our goal is to assist you as efficiently and effectively as we can.
Thanks and Best Regards,
Keith
Re: Root Partition is getting full every other day
Posted: Mon Mar 07, 2022 6:10 am
by G_Deshpande
Hi Keith
Thanks for revert. Certainly will do that.
We narrow down the issue....
Today as well consumed 100% of root partition and we went to location - /usr/local/nagios/var/nagios.log
So nagios.log file was consuming around 40 GB of disk, so we deleted old content of file.
So can you suggest from this situation that nagios.log is consuming more space, how can we get rid of this? if we resolve this then will not face disk crunch issue.
Regards
Gajanan
Re: Root Partition is getting full every other day
Posted: Mon Mar 07, 2022 5:58 pm
by kfanselow
Hi Gajanan,
That's what we'd like to find out. There's noticeable increase and we'd like to look a your logfiles outside of a semi-public forum thread.
Thanks and Best Regards,
Keith