No space left on device

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

tgriep wrote: Mon Jan 29, 2024 10:46 am The only way to see if you cleaned up the files that are taking up the inodes is to run this command again.

Code: Select all

{ find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }
Unfortunately that command does not give me any result.

root@RSB-VWA-T-MON:~# { find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -n; }
sort: cannot create temporary file in '/tmp': No space left on device
root@RSB-VWA-T-MON:~#
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: No space left on device

Post by tgriep »

The problem is that the command needs to create temporary files on the filesystem to create the list of inode usage but since there are not any free inodes, it cannot so that is why you see the error.

Login as root to the server and stop as many processes as you can.

Remove any log files or other files from the following folders

Code: Select all

/var/log
/var/log/httpd
/tmp
/var/tmp
Run the df -i command and if there is some free inodes, run the command and post the output.

Thank You.
Be sure to check out our Knowledgebase for helpful articles and solutions!
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

tgriep wrote: Tue Jan 30, 2024 4:38 pm The problem is that the command needs to create temporary files on the filesystem to create the list of inode usage but since there are not any free inodes, it cannot so that is why you see the error.

Login as root to the server and stop as many processes as you can.

Remove any log files or other files from the following folders

Code: Select all

/var/log
/var/log/httpd
/tmp
/var/tmp
Run the df -i command and if there is some free inodes, run the command and post the output.

Thank You.
I cleaned up all the files in the directories that ou mentioned and afterwards I was able to run the command with the below result:

414 /var/spool/postfix/etc/ssl/certs
448 /usr/share/mime/application
453 /usr/lib/firmware
459 /usr/share/consolefonts
491 /usr/share/man/man2
503 /usr/sbin
534 /usr/share/go-1.13/src/runtime
536 /usr/src/linux-raspi-headers-5.4.0-1068/include/uapi/linux
536 /usr/src/linux-raspi-headers-5.4.0-1070/include/uapi/linux
539 /usr/lib/firmware/amdgpu
563 /usr/include/linux
564 /usr/src/linux-raspi-headers-5.15.0-1042/include/uapi/linux
566 /usr/src/linux-raspi-headers-5.15.0-1037/include/uapi/linux
622 /usr/share/go-1.18/src/runtime
646 /usr/share/vim/vim82/syntax
681 /usr/share/go-1.18/src/cmd/go/testdata/script
744 /usr/local/nagios/nagios-4.4.9/html/angularjs/angular-1.3.9/i18n
759 /usr/share/python-babel-localedata/locale-data
903 /usr/share/bash-completion/completions
922 /usr/share/doc
981 /usr/share/man/man8
1045 /usr/lib/aarch64-linux-gnu
1235 /usr/bin
1294 /usr/src/linux-raspi-headers-5.4.0-1070/include/linux
1295 /usr/src/linux-raspi-headers-5.4.0-1068/include/linux
1309 /usr/share/man/man1
1393 /usr/src/linux-raspi-headers-5.15.0-1037/include/linux
1393 /usr/src/linux-raspi-headers-5.15.0-1042/include/linux
1443 /usr/src/linux-headers-5.4.0-1068-raspi/include/config
1443 /usr/src/linux-headers-5.4.0-1070-raspi/include/config
1453 /usr/share/go-1.13/test/fixedbugs
1760 /usr/share/go-1.18/test/fixedbugs
2545 /usr/share/man/man3
4031 /var/lib/dpkg/info
7750 /usr/src/linux-headers-5.15.0-1042-raspi/include/config
7750 /usr/src/linux-headers-5.15.0-1044-raspi/include/config
7752 /usr/src/linux-headers-5.15.0-1037-raspi/include/config
3512413 /usr/local/nagios/var/spool/nagfluxperfdata


So it is this directory "/usr/local/nagios/var/spool/nagfluxperfdata" that is taking up all the inodes. I tried to delete all the files in this directory but it gives me always this error. Can you help me out with this error?

root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/nagfluxperfdata# rm -f /usr/local/nagios/var/spool/nagfluxperfdata/*
bash: /usr/bin/rm: Argument list too long
root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/nagfluxperfdata# rm -f -r /usr/local/nagios/var/spool/nagfluxperfdata/*
bash: /usr/bin/rm: Argument list too long
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

This folder is full of .service and .host files. I am busy removing these files via the below commands.

find /usr/local/nagios/var/spool/nagfluxperfdata -name "*.service" -delete
find /usr/local/nagios/var/spool/nagfluxperfdata -name "*.host" -delete
cnorell
Developer
Posts: 65
Joined: Mon Nov 27, 2017 3:08 pm

Re: No space left on device

Post by cnorell »

When running a command with an asterisk, bash expands the asterisk to every matching file, and the linux kernel has a limit on how long a command can be. If you get more specific with the command so you match less files, it should work. Are the commands you last tried to run working?

Best Regards,

Cory Norell
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: No space left on device

Post by tgriep »

That is good news you found the files taking up the inodes.

Those files are for a application called Nagflux which is not a Nagios product so to find out why is failed and filled up the drive, you would have to go to their support site for help.


Thank You.
Be sure to check out our Knowledgebase for helpful articles and solutions!
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

cnorell wrote: Wed Jan 31, 2024 10:10 am When running a command with an asterisk, bash expands the asterisk to every matching file, and the linux kernel has a limit on how long a command can be. If you get more specific with the command so you match less files, it should work. Are the commands you last tried to run working?

Best Regards,

Cory Norell
Yes the 2 commands are working. I removed the .host files and it is busy now removing the .service files.

The amount of used inotes is decreasing now.

ubuntu@RSB-VWA-T-MON:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 999426 1481 997945 1% /run
/dev/mmcblk0p2 3754080 1460463 2293617 39% /
tmpfs 999426 1 999425 1% /dev/shm
tmpfs 999426 4 999422 1% /run/lock
/dev/mmcblk0p1 0 0 0 - /boot/firmware
tmpfs 199885 25 199860 1% /run/user/1000
ubuntu@RSB-VWA-T-MON:~$
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

The folder has been cleaned up and now the amount of used inodes is on 7%.

ubuntu@RSB-VWA-T-MON:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 999425 1494 997931 1% /run
/dev/mmcblk0p2 3754080 241529 3512551 7% /
tmpfs 999425 1 999424 1% /dev/shm
tmpfs 999425 4 999421 1% /run/lock
/dev/mmcblk0p1 0 0 0 - /boot/firmware
tmpfs 199885 25 199860 1% /run/user/1000


Running the below command does not give any errors anymore. I rebooted then the machine and I would suspect that Nagios is running again. I can see the Nagios processes are running again but unfortunately the web interface is still not accessible. Any idea how I get this working again? Did I delete too much?

ubuntu@RSB-VWA-T-MON:~$ sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.4.9
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 2022-11-16
License: GPL

Website: https://www.nagios.org
Reading configuration data...
Read main config file okay...
WARNING: The normal_check_interval attribute is deprecated and will be removed in future versions. Please use check_interval instead.
WARNING: The retry_check_interval attribute is deprecated and will be removed in future versions. Please use retry_interval instead.
Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Checked 526 services.
Checked 241 hosts.
Checked 16 host groups.
Checked 0 service groups.
Checked 4 contacts.
Checked 1 contact groups.
Checked 57 commands.
Checked 5 time periods.
Checked 0 host escalations.
Checked 0 service escalations.
Checking for circular paths...
Checked 241 hosts
Checked 0 service dependencies
Checked 0 host dependencies
Checked 5 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors: 0

Things look okay - No serious problems were detected during the pre-flight check
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: No space left on device

Post by tgriep »

Take a look at the Apache error logs in this folder /var/log/apache2 to see what errors are logged and use that to determine what to do to fix the issue.

Thank you.
Be sure to check out our Knowledgebase for helpful articles and solutions!
ptran
Posts: 35
Joined: Fri Jan 12, 2024 5:50 am

Re: No space left on device

Post by ptran »

tgriep wrote: Thu Feb 01, 2024 10:06 am Take a look at the Apache error logs in this folder /var/log/apache2 to see what errors are logged and use that to determine what to do to fix the issue.

Thank you.
This folder does not exist.

root@RSB-VWA-T-MON:/var/log# ls -ls
total 944
28 -rw-r----- 1 syslog adm 23362 Feb 1 16:13 auth.log
4 -rw-r----- 1 syslog adm 477 Feb 1 08:08 auth.log.1
4 -rw-rw---- 1 root utmp 1200 Feb 1 08:08 btmp
0 -rw-rw---- 1 root utmp 0 Mar 20 2023 btmp.1
4 -rw-r----- 1 root adm 3274 Feb 1 08:08 cloud-init-output.log
76 -rw-r----- 1 syslog adm 76520 Feb 1 08:08 cloud-init.log
52 -rw-r----- 1 root adm 50856 Feb 1 08:08 dmesg
4 -rw-r----- 1 syslog adm 1857 Feb 1 08:08 kern.log
64 -rw-r----- 1 syslog adm 61471 Feb 1 08:08 kern.log.1
4 -rw-rw-r-- 1 root utmp 296296 Feb 1 08:09 lastlog
124 -rw-r----- 1 syslog adm 123396 Feb 1 16:11 mail.log
4 drwx------ 2 root root 4096 Mar 20 2023 private
4 -rw-r--r-- 1 root root 2727 Feb 1 08:09 smsd.log
452 -rw-r----- 1 syslog adm 461481 Feb 1 16:12 syslog
100 -rw-r----- 1 syslog adm 102202 Feb 1 08:08 syslog.1
4 drwxr-xr-x 2 root root 4096 Feb 1 08:08 unattended-upgrades
4 -rw-r--r-- 1 root root 500 Feb 1 08:08 usb_modeswitch_1-1.1
4 -rw-r----- 1 root adm 1665 Feb 1 16:11 vsftpd.log
4 -rw------- 1 root root 1272 Mar 27 2023 vsftpd.log.1
4 -rw-rw-r-- 1 root utmp 2800 Feb 1 08:09 wtmp
root@RSB-VWA-T-MON:/var/log# ²
Post Reply