No space left on device

[ptran]

It looks like the

Code: Select all

/dev/mmcblk0p2

directory is out of inodes based on your output posted. Is that the partition Nagios is installed on? If so, I don't know of a way to add inodes without remaking the filesystem.

Best Regards

Cory Norell

How can I check that nagios is installed on this portion /dev/mmcblk0p2?

[swolf]

It looks like the

Code: Select all

/dev/mmcblk0p2

directory is out of inodes based on your output posted. Is that the partition Nagios is installed on? If so, I don't know of a way to add inodes without remaking the filesystem.

Best Regards

Cory Norell

How can I check that nagios is installed on this portion /dev/mmcblk0p2?

Hi @ptran - the way you'd check on your own is by looking at the paths that show in your `df -h` output, and comparing them to the paths where Nagios XI is installed. For your particular output, you basically have everything on that one partition, so you're good to go. I would look at the paths that @tgriep posted in his last reply - let us know if that gets you anywhere useful.

[ptran]

The inodes being full on the drive is the issue. See this when you ran the df -i command.

/dev/mmcblk0p2 3754080 3754080 0 100% /

When the inodes are full, no new files can be created which is what you are having.

Some where on the drive is a folder with a lot of files, (thousands to million of files) and they need to be cleaned up.

Look in the following folders and if you see a lot of files, they can be deleted.

/usr/local/nagios/var/spool/perfdata/
/usr/local/nagios/var/spool/xidpe/
/usr/local/nagios/var/spool/checkresults
/var/spool/snmptt

Thank You.

This is the result. Not much files to be found back in those folders (if they exist) to clean up.

Last login: Mon Jan 22 11:21:34 2024 from 192.168.3.43
ubuntu@RSB-VWA-T-MON:~$ sudo su
sudo: unable to resolve host RSB-VWA-T-MON: Temporary failure in name resolution
root@RSB-VWA-T-MON:/home/ubuntu# cd /usr/local/nagios/var/spool/perfdata/
bash: cd: /usr/local/nagios/var/spool/perfdata/: No such file or directory
root@RSB-VWA-T-MON:/home/ubuntu# cd /usr/local/nagios/var/spool/perfdata
bash: cd: /usr/local/nagios/var/spool/perfdata: No such file or directory
root@RSB-VWA-T-MON:/home/ubuntu# cd /usr/local/nagios/var/spool/xidpe
bash: cd: /usr/local/nagios/var/spool/xidpe: No such file or directory
root@RSB-VWA-T-MON:/home/ubuntu# cd /usr/local/nagios/var/spool/checkresults
root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/checkresults# ls -ls
total 0
root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/checkresults# cd /var/spool/snmptt
bash: cd: /var/spool/snmptt: No such file or directory
root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/checkresults#

[tgriep]

Here is a web site that has examples of how to find out which folder has the most inodes on the server. See if one of the examples will help you find them.
https://unix.stackexchange.com/question ... being-used

To run those commands, you need to login as root.

Thank You.

[cnorell]

Another thing you can try: run the following command to find the directories with the most inode usage:

Code: Select all

du --inodes | sort -rh | head -15

The above example will show 15 results. You can change the argument passed to head (

Code: Select all

-10

) to view a different number of results.

Remember you will need to run the above command as root to ensure you can "see" into the entire filesystem. You can then look into those directories to see if anything can be cleaned up.

Best Regards,

Cory Norell

[ptran]

Another thing you can try: run the following command to find the directories with the most inode usage:

Code: Select all

du --inodes | sort -rh | head -15

The above example will show 15 results. You can change the argument passed to head (

Code: Select all

-10

) to view a different number of results.

Remember you will need to run the above command as root to ensure you can "see" into the entire filesystem. You can then look into those directories to see if anything can be cleaned up.

Best Regards,

Cory Norell

This is the result. What can I do with this then?

root@RSB-VWA-T-MON:/proc# du --inodes | sort -rh | head -15
du: cannot access './54481/task/54481/fd/4': No such file or directory
du: cannot access './54481/task/54481/fdinfo/4': No such file or directory
du: cannot access './54481/fd/3': No such file or directory
du: cannot access './54481/fdinfo/3': No such file or directory
67988 .
1581 ./53121
1553 ./54445
1378 ./54445/task
1329 ./454
1212 ./721
1107 ./sys
1072 ./454/task
930 ./807
866 ./718
845 ./sys/net
826 ./721/task
806 ./53121/task
727 ./1
714 ./717

[ptran]

Here is a web site that has examples of how to find out which folder has the most inodes on the server. See if one of the examples will help you find them.
https://unix.stackexchange.com/question ... being-used

To run those commands, you need to login as root.

Thank You.

Here are the results of the commands found in this link:

root@RSB-VWA-T-MON:/proc/53121# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null
root@RSB-VWA-T-MON:/proc/53121# { find / -xdev -printf '%h\0' |sort -z |uniq -zc |sort -zk1rn; } 2>/dev/null
root@RSB-VWA-T-MON:/proc/53121# for i in `find . -type d `; do echo `ls -a $i | wc -l` $i; done | sort -n
3 ./attr/smack
3 ./net/ipconfig
3 ./net/netfilter
3 ./task/53121/attr/smack
3 ./task/53121/net/ipconfig
3 ./task/53121/net/netfilter
3 ./task/53122/attr/smack
3 ./task/53122/net/ipconfig
3 ./task/53122/net/netfilter
3 ./task/53123/attr/smack
3 ./task/53123/net/ipconfig
3 ./task/53123/net/netfilter
3 ./task/53124/attr/smack
3 ./task/53124/net/ipconfig
3 ./task/53124/net/netfilter
3 ./task/53125/attr/smack
3 ./task/53125/net/ipconfig
3 ./task/53125/net/netfilter
5 ./attr/apparmor
5 ./net/dev_snmp6
5 ./net/stat
5 ./task/53121/attr/apparmor
5 ./task/53121/net/dev_snmp6
5 ./task/53121/net/stat
5 ./task/53122/attr/apparmor
5 ./task/53122/net/dev_snmp6
5 ./task/53122/net/stat
5 ./task/53123/attr/apparmor
5 ./task/53123/net/dev_snmp6
5 ./task/53123/net/stat
5 ./task/53124/attr/apparmor
5 ./task/53124/net/dev_snmp6
5 ./task/53124/net/stat
5 ./task/53125/attr/apparmor
5 ./task/53125/net/dev_snmp6
5 ./task/53125/net/stat
7 ./task
12 ./attr
12 ./ns
12 ./task/53121/attr
12 ./task/53121/ns
12 ./task/53122/attr
12 ./task/53122/ns
12 ./task/53123/attr
12 ./task/53123/ns
12 ./task/53124/attr
12 ./task/53124/ns
12 ./task/53125/attr
12 ./task/53125/ns
16 ./fd
16 ./fdinfo
16 ./task/53121/fd
16 ./task/53121/fdinfo
16 ./task/53122/fd
16 ./task/53122/fdinfo
16 ./task/53123/fd
16 ./task/53123/fdinfo
16 ./task/53124/fd
16 ./task/53124/fdinfo
16 ./task/53125/fd
16 ./task/53125/fdinfo
45 ./task/53121
45 ./task/53122
45 ./task/53123
45 ./task/53124
45 ./task/53125
52 .
59 ./net
59 ./task/53121/net
59 ./task/53122/net
59 ./task/53123/net
59 ./task/53124/net
59 ./task/53125/net
610 ./map_files

[cnorell]

Hmm, I expected to see some directories with hundreds of thousands of files in the output of the command I suggested. The goal of that command was to find a few directories that had inordinate amounts of inodes used to see where we could start clean up. Let me do some more digging and see if I can find another potential solution.

Best Regards,

Cory Norell

[tgriep]

You ran the commands from the wrong folder. They were ran from the /proc/53121 folder which is not correct.

Change to the root folder of the drive by running this

Code: Select all

cd /

Then run the commands or better yet, run this command that starts at the root folder and will show 30 of the largest folders with used inodes.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null | tail -30

Thank you.

[ptran]

You ran the commands from the wrong folder. They were ran from the /proc/53121 folder which is not correct.

Change to the root folder of the drive by running this

Code: Select all

cd /

Then run the commands or better yet, run this command that starts at the root folder and will show 30 of the largest folders with used inodes.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null | tail -30

Thank you.

I ran the command but it does not give me any result, is this normal?

ubuntu@RSB-VWA-T-MON:~$ sudo su
sudo: unable to resolve host RSB-VWA-T-MON: Temporary failure in name resolution
root@RSB-VWA-T-MON:/home/ubuntu# cd\
>
root@RSB-VWA-T-MON:~# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; } 2>/dev/null | tail -30
root@RSB-VWA-T-MON:~#