Nagios Support Forum

"Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: Root Partition
Host: localhost
State: WARNING
Info:
DISK WARNING - free space: / 6901 MiB (18.23% inode=78%):
Date/Time: 03/11/2025 23:37:30"

I need to clear space on my Nagios VM, I have tried following other guides in this forum but none seem to be working. It very well could be user error, given that I dont even know how to copy the CLI to paste it here lol.

Hi @Tate,

Depending on your configuration, excess logs can be one of the issues that'll cause your VM to fill up its disk, but there are a few others as well. First of all, how much space do you have allocated to this VM? You can check this by typing in a terminal and adding up the values in the "Size" column. The "df" tool can help you drill down into your file system to figure out which directories may be the issue. We recommend a minimum of 40GB allocated to your VM, or available as physical storage if you're not running in a virtual environment.

We have additional suggestions in this knowledge base article for addressing some of the other issues that can cause your disk to fill up. I'd recommend taking a look at the backup directory as well as your mysql directories, per the suggestions in the article as that seems more likely to be your issue than logs, particularly if log rotation hasn't helped. The article I linked above also contains another link to a guide for resizing your VM, if it turns out the issue is that you simply need more space available.

Please let us know if you need more help looking into this. If so, please let us know what version of XI you're on, and what Linux distro/version you have your XI installed on. This will help with providing correct commands as well as general context moving forward.

Here is how much space I have allocated, it is over 40Gigs.

What commands would I use for finding the info to answer these questions? "

jsimon wrote: ↑Wed Mar 12, 2025 9:48 am If so, please let us know what version of XI you're on, and what Linux distro/version you have your XI installed on. This will help with providing correct commands as well as general context moving forward.

"

Also the VM seems to be running Red Hat Enterprise Linux 9

NagiosXI version = 2024R1.01

Ok so I accessed this forum post https://support.nagios.com/kb/article.php?id=26 and checked the mysql and postgresql directories: I also checked the 10 largest files under /var/log/ directory:

Taking a look at the last post you made, the first screenshot you posted is of your crontab fragment. This controls scheduled scripts that run for Nagios XI, but doesn't really tell us anything notable about file sizes on your system. The reason that was posted in the article you're referencing is that the article suggests disabling database backup jobs if you don't want them to run, and the crontab is how you'd do that.

For now, try running this in your terminal on your XI machine: This is the same command you ran to get the largest 10 files in /var/log, but it's referencing the /store/backups/nagiosxi location. In addition, I added the "h" parameter to the "du -a" command at the start. This gives us slightly more human readable size values. This will let us know how much space your backups are taking up. In addition, it may be worth revisiting the steps you took to rotate logs. It looks like you're using about 2GB for your logs, which isn't huge -- about 5% of your total disk allocation. This could be worth rotating but isn't likely to be the major source of storage issues.

You can run the above command, changing out the directory location "/store/backups/nagiosxi" to get an idea of how much space individual directory locations are taking up on your system. You might find it helpful to run that command on the various directories mentioned in the KB article I previously linked to get an idea of what you need to look at rotating

Currently we are losing about a gigabyte of space everyday according to the Nagios Alerts, I went through using
du -ah /store/backups/nagiosxi | sort -n -r | head -n 10
and went through each of these directories: /afs /boot /dev /etc /lib /media /opt /root /sbin /store /tmp /var /bin /downloads /home /lib64 /mnt /proc /run /srv /sys /usr

None of them had even multiple files that were larger than 1.5G, frankly I am lost as to where I am losing a Gig a day. Also I a system-journal message keeps popping up

[root@Nagios mapper]# [271433.117597] systemd-journald[611]: Data hash table of /run/log/journal/649b351f4e27490f82204962976f643 e/system.journal has a fill level at 75.0 (20228 of 26965 items, 8388608 file size, 414 bytes per hash table item), suggesting rotation.
[271433.117597] systemd-journald[611]: /run/log/journal/649b351f4e27490f82204962976f643e/system.journal: Journal header limits reached or header out-of-date, rotating.

When you ran the command, were you changing directories and running the same command in each, or modifying the directory in the command? The system journal you mention is referencing a lot of space being used within the /run/log/journal directory, so I find it surprising you weren't seeing anything over 1.5 if you were running a command like this:
If you navigated into different directories and then ran the exact command: The command would be providing du output for only the /store/backups/nagiosxi directory every time.

To go a bit more in depth on what we're actually doing here, the "du" command provides disk-usage for files and directories within the directory you tell it to look in. The following piped "sort" and "head" commands are sorting the output of this command by file size (descending), and then printing the top 10 results.

Otherwise, from the journal message you're getting, I would suggest doing some research into journald configuration and rotation. It might be that your system is logging too much information on a regular basis for your storage to account for.

I see, I must have ran the command wrong.

I think I found the issue though, SNMPTT service is running (Active: active), but SNMP traps are piling up in /var/spool/snmptt, consuming 17GB of space and not processing traps. Also the Memory Usage is High (1.3GB) from SNMPTT. I did run into an Issue: "Argument list too long" When Deleting Files. My /var/spool/snmptt/ directory contains too many files for rm -rf to handle in a single command. Apparently this happens when the number of files exceeds the system's argument limit.

I am running rm -rf /var/spool/snmptt/* but I get -bash: /usr/bin/rm: Argument list too long so from googling and chat Gpt the solution seems to be this: find /var/spool/snmptt/ -type f -delete which is currently running and I am waiting to see what happens...

It definitely sounds like this might be the cause of your issue! Clearing out the stuck traps will help but you'll also need to figure out why they aren't being processed, or stop them from being sent. When you check the status of the SNMPTT service, do you see any errors listed?