/var running Out Of Disk Space On The Nagios XI VM

[vappukuttan]

Hello,

I have reviewed documentation on tuning Nagios XI and some of the support threads like
https://support.nagios.com/kb/article/n ... vm-26.html
https://support.nagios.com/forum/viewto ... 16&t=47171

I have Nagios XI 5.6.9 running. We have been running into issue where /var fills up. The /var volume is 10GB. Today it happened 3-4 times. Once I run the repair the volume goes down to 50% used.

When there is the issue and i look at the files linked to the mysqld process, i do see two large files under /var/tmp that are in (deleted) status. Based on the size they show, it added up to 3.5GB

mysqld 12774 28749 mysql 52u REG 253,3 1815504285 16840828 /var/tmp/STkzFYxM (deleted)
mysqld 12774 28749 mysql 95u REG 253,3 1733230592 16840829 /var/tmp/STa872Nk (deleted)

Also like other obvious threads I do see the nagios_logentries taking up another 3.1GB. I have just 10 days for "Max Log Entries Age:" within the Performance settings databases NDO Database configuration screen.

-rw-rw----. 1 mysql mysql 2.2G Jan 22 14:00 nagios_logentries.MYD
-rw-rw----. 1 mysql mysql 921M Jan 22 18:26 nagios_logentries.MYI


Any thoughts on what I should be checking or tuning.

Thank you for any help or advice,

[Box293]

You disks should be larger, there are links in that KB article for increasing the disk space of the VM.

I'm sure there's some tweaking that could be done to reduce disk space but honestly by the time you do that, along with trial and errors you could have increased the disk space and forgotten about it.

[vappukuttan]

Thank you for the quick response. Is there a recommended size for the /var or a way to come up with it. We are in the process of moving from Icinga and in that world it is not disk, memory or cpu intensive for the same amount of monitors we have within Nagios XI. So it caught us by surprise when the Nagios XI system makes 4GB vanish in a few minutes

[vappukuttan]

I posted something about 10 mins ago and it still hasnt showed up, not sure if it didnt get submitted.

We are moving from Icinga to Nagios XI. Both running almost the same amount of monitors. The Icinga server as a whole only has 40GB disk space. The Nagios XI has the following.

Code: Select all

Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/OS_VG-root_lv   33G   24G  9.8G  71% /
devtmpfs                   7.8G     0  7.8G   0% /dev
/dev/mapper/OS_VG-tmp_lv   5.0G  110M  4.9G   3% /tmp
/dev/mapper/OS_VG-home_lv   10G  332M  9.7G   4% /home
/dev/mapper/OS_VG-var_lv    10G  4.8G  5.3G  48% /var
/dev/sda1                 1014M  300M  715M  30% /boot

Is there a recommended size for /var or is there a method to calculate it.

Thank you again

[vappukuttan]

Update:
Kept an eye on the system.. There was the "database maintenance" showing red on the Nagios XI Admin page. So during the time it tries to run that.. there are temp files ST* created by the mysqld process which is as large as the database or close to the data its working with. I moved the tmpdir to a different volume with space.. and i saw the temp files created by the process go up to 3.6GB (2 files of 1.8GB each), then it disappeared .. and once I saw the ST* temp files clear out.. I saw the "database maintenance" switch from red to green.

Thank you,

[scottwilkerson]

Update:
Kept an eye on the system.. There was the "database maintenance" showing red on the Nagios XI Admin page. So during the time it tries to run that.. there are temp files ST* created by the mysqld process which is as large as the database or close to the data its working with. I moved the tmpdir to a different volume with space.. and i saw the temp files created by the process go up to 3.6GB (2 files of 1.8GB each), then it disappeared .. and once I saw the ST* temp files clear out.. I saw the "database maintenance" switch from red to green.

Thank you,

Great!

May we mark the case resolved?

[vappukuttan]

Hello Scott,

I was hoping someone would give a norm on what the disk space sizes that are required for Nagios XI.

I saw an article where it said nodes=50, services=250 - 40GB; nodes=100, services=500 - 80GB; nodes > 500, services >2500 - 120GB.
But was vague on explaining what the /var volume space should be. Also 120GB for NagiosXI if you have >2500 services seemed a very high number.

Thank you,

[scottwilkerson]

Hello Scott,

I was hoping someone would give a norm on what the disk space sizes that are required for Nagios XI.

I saw an article where it said nodes=50, services=250 - 40GB; nodes=100, services=500 - 80GB; nodes > 500, services >2500 - 120GB.
But was vague on explaining what the /var volume space should be. Also 120GB for NagiosXI if you have >2500 services seemed a very high number.

Thank you,

What you read is our companies baseline suggestion for installs with the bulk of the data ending up in /usr & /var, but every organizations mileage will vary because it depends on many factors including hosts/services if they all return performance data, frequency of checks, etc., etc.

[vappukuttan]

Hello,

I understand.. but some rules would help to get an idea if we are not off the charts..

So if the specs are something like the below, what would you recommend the /var volume size :

50 servers
50x100 = 5000 services
Yes on returning performance data on 50% of the services
frequency is check every 5 minutes
(within Database Performance - Nagios XI database)
Max Audit log age = 30 days
(within Database Performance - NDO database)
Max days to keep logs = 10 days
Max Notification age = 30 days
Max State History age = 100 days
Max comment History age = 730 mins
Optimize Interval = 60 mins



Thank you,

[scottwilkerson]

I would think 50G-75G should be enough