Page 1 of 1
Ran out of inodes
Posted: Tue Mar 10, 2015 2:24 pm
by lavignp
My NagiosXI server is currently down, and it seems it is because I am out of inodes.
I am on CentOS 6.3, Kernel 2.6.32-431.11.2.e16.x86_64
I am running it from the VM image
What happened was, I apparently ran out of disk space (not properly cleaning out backups created by backup_xi.sh)
I manged to clean the old backups out, and got the partition back down to only 75% used.
But, when I tried to start httpd, I got:
Code: Select all
touch: cannot touch `/var/lock/subsys/httpd': No space left on device
There was plenty of disk space, so then I realized that the inodes might be full up. Sure enough:
Code: Select all
[root@v-nagios01 /]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
6529024 6529024 0 100% /
tmpfs 1006700 1 1006699 1% /dev/shm
/dev/sda1 25688 46 25642 1% /boot
This is where I have hit my road block. How in the world can I free up these inodes?
thanks
-Paul
Re: Ran out of inodes
Posted: Tue Mar 10, 2015 2:29 pm
by jdalrymple
inodes is a filesystem construct and unfortunately the number cannot be increased like filesystem size can be.
The way to free inodes is to simply delete files. The question is where is it that you have all these inodes in use, 6.5M is a LOT of inodes. To find where they're all being consumed run this on your server:
Code: Select all
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n
If you have any NFS mounts or anything we may want to rewrite that command to exclude them - let us know.
Re: Ran out of inodes
Posted: Tue Mar 10, 2015 5:19 pm
by lavignp
Thanks. I had forgotten to mention that I had tried that exact command, but because I was 100% burned up on inodes, I couldn't run commands like that because it couldn't even create anything in /tmp.
I went into /tmp, and manually removed as much as I could, which then gave me the room to find out that the culprit was in /var/spool/snmptt.
I am still removing files from there right now.
Next, I have to figure out why I keep having problems with the snmp traps. This kind of thing has happened to me before, although, not to the point of consuming all free space and inodes.
Re: Ran out of inodes
Posted: Tue Mar 10, 2015 5:52 pm
by Box293
So to address your /var/spool/snmptt directory filling up.
You could create a service that watches the snmptt service. Here's an example for watching the mysqld service:
Requires:
check_init_service plugin which is distributed as part of linux-nrpe-agent.tar.gz
http://assets.nagios.com/downloads/nagi ... ent.tar.gz
Requires the following entries in /etc/sudoers
Code: Select all
Defaults:NAGIOSXI !requiretty
NAGIOSXI ALL=NOPASSWD: /usr/local/nagios/libexec/check_init_service
define command {
command_name check_service_status
command_line $ARG1$ $USER1$/check_init_service $ARG2$
}
define service {
host_name localhost
service_description Service Status - mysqld
use local-service
check_command check_service_status!sudo!mysqld!!!!!!
register 1
}
General service output is:
mysqld (pid 2469) is running...
You could create a service that watches the amount of files in the spool directory. Here's an example for watching the xidpe spool directory:
Requires:
http://exchange.nagios.org/directory/Pl ... nt/details
NOTE: The plugin needs to be gunziped and then untarred
Code: Select all
gunzip check_file_count.tar.gz
tar xf check_file_count.tar
define command {
command_name check_file_count
command_line $USER1$/check_file_count $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$ $ARG7$ $ARG8$
}
define service {
host_name localhost
service_description File Count - xidpe Spool Directory
use local-service
check_command check_file_count!-t=file!-s=no!-wt=high!-w 500!-c 1000!-p=yes!/usr/local/nagios/var/spool/xidpe/!
register 1
}
General service output is:
OK: Number of files in /usr/local/nagios/var/spool/xidpe/ is 0 (Warn=500, Crit=1000)
Finally, what is the output of:
Re: Ran out of inodes
Posted: Tue Mar 10, 2015 6:06 pm
by lavignp
Cool idea, good to know. Thanks.
The output of /var/spool is currently:
Code: Select all
[root@v-nagios01 spool]# ls -al /var/spool/
total 361880
drwxr-xr-x. 13 root root 4096 Mar 11 2014 .
drwxr-xr-x. 21 root root 4096 Mar 12 2014 ..
drwxr-xr-x. 2 abrt abrt 4096 Jan 13 08:33 abrt
drwx------. 2 abrt abrt 4096 Nov 22 2013 abrt-upload
drwxr-xr-x. 2 root root 4096 Nov 23 2013 anacron
drwx------. 3 daemon daemon 4096 Jan 30 2012 at
drwx------. 2 root root 4096 Nov 23 2013 cron
drwxr-xr-x. 2 root root 4096 Sep 23 2011 lpd
drwxrwxr-x. 2 root mail 4096 Mar 7 15:50 mail
drwxr-xr-x 2 nagios nagios 4096 Aug 30 2013 nagios
drwxr-xr-x. 2 root root 4096 Mar 10 11:57 plymouth
drwxr-xr-x. 16 root root 4096 Feb 20 2014 postfix
drwxrwxr-x 2 snmptt snmptt 370491392 Mar 10 16:04 snmptt
Re: Ran out of inodes
Posted: Tue Mar 10, 2015 6:08 pm
by Box293
The snmptt spool directory permissions appear to be correct.