No space left on device

[ptran]

Hmm, I expected to see some directories with hundreds of thousands of files in the output of the command I suggested. The goal of that command was to find a few directories that had inordinate amounts of inodes used to see where we could start clean up. Let me do some more digging and see if I can find another potential solution.

Best Regards,

Cory Norell

I ran the same commands now from the root folder but it does not give more useful info.

root@RSB-VWA-T-MON:~# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
root@RSB-VWA-T-MON:~# { find / -xdev -printf '%h\0' |sort -z |uniq -zc |sort -zk1rn; } 2>/dev/null
root@RSB-VWA-T-MON:~# for i in `find . -type d `; do echo `ls -a $i | wc -l` $i; done | sort -n
2 ./snap/lxd/23999
2 ./snap/lxd/24065
2 ./snap/lxd/common
3 ./.cache
3 ./.cache/snowflake
3 ./.local
3 ./.local/share
3 ./.local/share/nano
3 ./.ssh
3 ./snap
6 ./snap/lxd
12 .
root@RSB-VWA-T-MON:~#

[cnorell]

As a long shot question - are you running your XI server on bare metal, or are you in a containerized environment?

Best Regards,

Cory Norell

[ptran]

As a long shot question - are you running your XI server on bare metal, or are you in a containerized environment?

Best Regards,

Cory Norell

I do not know what you mean exactly with bare metal but it is not running on a server machine but a small machine/chip card with a 64GB SD card as the hard drive.

Best regards.

[tgriep]

The command I posted is supposed to output data but is maybe failing because of the inode issue and not displaying data.

Login as root and run this shortened command to see if it works.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }

[ptran]

The command I posted is supposed to output data but is maybe failing because of the inode issue and not displaying data.

Login as root and run this shortened command to see if it works.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }

No result for this command:

root@RSB-VWA-T-MON:~# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }
sort: cannot create temporary file in '/tmp': No space left on device
root@RSB-VWA-T-MON:~#

[cnorell]

If you're not getting output because of a lack of free inodes, maybe deleting some files you don't need right before running the command will help?

Best Regards,

Cory Norell

[ptran]

If you're not getting output because of a lack of free inodes, maybe deleting some files you don't need right before running the command will help?

Best Regards,

Cory Norell

I have cleaned up a bunch of files already but apparently they were not part of these inode files. How can I see which files are part of these inodes?

[cnorell]

Every file on the filesystem will have an associated inode, so deleting any file should free up the associated inode. That said, sometimes files aren't actually deleted until the process that was using the file releases the file.

If you've tried deleting files and then restarting the system to force release the files, the only thing I can think of is that inodes are being created as quickly as you're deleting them. Perhaps try stopping the Nagios process, then deleting files so you can get output from the command Tom recommended?

As an aside, one possible problem directory is

Code: Select all

/var/lib/php/session

. Sometimes PHP sessions aren't cleaned up properly, and are relatively safe to delete to make room for command output.

Best Regards,

Cory Norell

[tgriep]

The only way to see if you cleaned up the files that are taking up the inodes is to run this command again.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }

[ptran]

Every file on the filesystem will have an associated inode, so deleting any file should free up the associated inode. That said, sometimes files aren't actually deleted until the process that was using the file releases the file.

If you've tried deleting files and then restarting the system to force release the files, the only thing I can think of is that inodes are being created as quickly as you're deleting them. Perhaps try stopping the Nagios process, then deleting files so you can get output from the command Tom recommended?

As an aside, one possible problem directory is

Code: Select all

/var/lib/php/session

. Sometimes PHP sessions aren't cleaned up properly, and are relatively safe to delete to make room for command output.

Best Regards,

Cory Norell

There are no files in /var/lib/php/sessions.

ubuntu@RSB-VWA-T-MON:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 999426 1486 997940 1% /run
/dev/mmcblk0p2 3754080 3754080 0 100% /
tmpfs 999426 1 999425 1% /dev/shm
tmpfs 999426 4 999422 1% /run/lock
/dev/mmcblk0p1 0 0 0 - /boot/firmware
tmpfs 199885 25 199860 1% /run/user/1000

Does this mean there are 3754080 files? But there is no way to list the files that are linked to the files system /dev/mmcblk0p2?