Ran out of inodes

[lavignp]

My NagiosXI server is currently down, and it seems it is because I am out of inodes.

I am on CentOS 6.3, Kernel 2.6.32-431.11.2.e16.x86_64
I am running it from the VM image

What happened was, I apparently ran out of disk space (not properly cleaning out backups created by backup_xi.sh)
I manged to clean the old backups out, and got the partition back down to only 75% used.
But, when I tried to start httpd, I got:

Code: Select all

touch: cannot touch `/var/lock/subsys/httpd': No space left on device

There was plenty of disk space, so then I realized that the inodes might be full up. Sure enough:

Code: Select all

[root@v-nagios01 /]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/VolGroup00-LogVol00
                     6529024 6529024       0  100% /
tmpfs                1006700       1 1006699    1% /dev/shm
/dev/sda1              25688      46   25642    1% /boot

This is where I have hit my road block. How in the world can I free up these inodes?

thanks
-Paul

[jdalrymple]

inodes is a filesystem construct and unfortunately the number cannot be increased like filesystem size can be.

The way to free inodes is to simply delete files. The question is where is it that you have all these inodes in use, 6.5M is a LOT of inodes. To find where they're all being consumed run this on your server:

Code: Select all

find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n

If you have any NFS mounts or anything we may want to rewrite that command to exclude them - let us know.

[lavignp]

Thanks. I had forgotten to mention that I had tried that exact command, but because I was 100% burned up on inodes, I couldn't run commands like that because it couldn't even create anything in /tmp.

I went into /tmp, and manually removed as much as I could, which then gave me the room to find out that the culprit was in /var/spool/snmptt.

I am still removing files from there right now.

Next, I have to figure out why I keep having problems with the snmp traps. This kind of thing has happened to me before, although, not to the point of consuming all free space and inodes.

[Box293]

So to address your /var/spool/snmptt directory filling up.

You could create a service that watches the snmptt service. Here's an example for watching the mysqld service:

Requires:
check_init_service plugin which is distributed as part of linux-nrpe-agent.tar.gz
http://assets.nagios.com/downloads/nagi ... ent.tar.gz

Requires the following entries in /etc/sudoers

Code: Select all

Defaults:NAGIOSXI !requiretty
NAGIOSXI ALL=NOPASSWD: /usr/local/nagios/libexec/check_init_service


define command {
       command_name check_service_status
       command_line $ARG1$ $USER1$/check_init_service $ARG2$
}

define service {
    host_name localhost
    service_description Service Status - mysqld
    use local-service
    check_command check_service_status!sudo!mysqld!!!!!!
    register 1
    }

General service output is:
mysqld (pid 2469) is running...


You could create a service that watches the amount of files in the spool directory. Here's an example for watching the xidpe spool directory:

Requires:
http://exchange.nagios.org/directory/Pl ... nt/details

NOTE: The plugin needs to be gunziped and then untarred

Code: Select all

gunzip check_file_count.tar.gz
tar xf check_file_count.tar

define command {
       command_name check_file_count
       command_line $USER1$/check_file_count $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$ $ARG7$ $ARG8$
}

define service {
    host_name localhost
    service_description File Count - xidpe Spool Directory
    use local-service
    check_command check_file_count!-t=file!-s=no!-wt=high!-w 500!-c 1000!-p=yes!/usr/local/nagios/var/spool/xidpe/!
    register 1
    }

General service output is:
OK: Number of files in /usr/local/nagios/var/spool/xidpe/ is 0 (Warn=500, Crit=1000)


Finally, what is the output of:

Code: Select all

ls -al /var/spool/

[lavignp]

Cool idea, good to know. Thanks.

The output of /var/spool is currently:

Code: Select all

[root@v-nagios01 spool]# ls -al /var/spool/
total 361880
drwxr-xr-x. 13 root   root        4096 Mar 11  2014 .
drwxr-xr-x. 21 root   root        4096 Mar 12  2014 ..
drwxr-xr-x.  2 abrt   abrt        4096 Jan 13 08:33 abrt
drwx------.  2 abrt   abrt        4096 Nov 22  2013 abrt-upload
drwxr-xr-x.  2 root   root        4096 Nov 23  2013 anacron
drwx------.  3 daemon daemon      4096 Jan 30  2012 at
drwx------.  2 root   root        4096 Nov 23  2013 cron
drwxr-xr-x.  2 root   root        4096 Sep 23  2011 lpd
drwxrwxr-x.  2 root   mail        4096 Mar  7 15:50 mail
drwxr-xr-x   2 nagios nagios      4096 Aug 30  2013 nagios
drwxr-xr-x.  2 root   root        4096 Mar 10 11:57 plymouth
drwxr-xr-x. 16 root   root        4096 Feb 20  2014 postfix
drwxrwxr-x   2 snmptt snmptt 370491392 Mar 10 16:04 snmptt

[Box293]

The snmptt spool directory permissions appear to be correct.