No space left on device

[kg2857]

The root user is the root user. It's called root.
whoami

[ptran]

The root user is the root user. It's called root.
whoami

I only have on my side 2 accounts: ubuntu and nagios. I don't have any pwd available for root.

I tried logging in with root and pwd = nagiosxi but that does not seem to be the pwd.

[kg2857]

If you can't be root you need to turn this over to someone who can be root. Otherwise this is pointless.
This is basic linux administration.
When someone with root access and admin experience can respond a meaningful reply might be forthcoming.

[cnorell]

Yeah, unfortunately you're going to need to run it up the ladder in your organization and determine who has root access. Without root, there's simply no other way to modify those directories or the contents within.

Once you get root access, post back here and we'll see if we can make some more progress.

Best Regards,

Cory Norell

[ptran]

Yeah, unfortunately you're going to need to run it up the ladder in your organization and determine who has root access. Without root, there's simply no other way to modify those directories or the contents within.

Once you get root access, post back here and we'll see if we can make some more progress.

Best Regards,

Cory Norell

I found out that when using the command "sudo su" you can switch to the root account. Afterwards I could delete the files that I could not delete in the past.

I went into the folder /tmp and removed everything in there via the command "rm -r *"
I went into the folder /var/log/journal/db1a06b71615441ca77b6fa011cac336 and delete there also all the journal files in there.
I went into the folder /var/log and delete there the syslog.1 file which was about 700MB.

All of this should have provided me a lot of free diskspace but after reboot of the system, it still complains of no space left on device. Are the deleted files kept in a bin somewhere that I need to clear out first before it is definately removed from the system?

I tried changing the pwd of the root account via "passwd root" but it gives me below error then.

ubuntu@RSB-VWA-T-MON:~$ sudo su
root@RSB-VWA-T-MON:/home/ubuntu# cd /var/log/journal/db1a06b71615441ca77b6fa011cac336^C
root@RSB-VWA-T-MON:/home/ubuntu# passwd root
New password:
Retype new password:
passwd: Authentication token manipulation error
passwd: password unchanged
root@RSB-VWA-T-MON:/home/ubuntu#

This is the result of the du command on the /tmp folder:
root@RSB-VWA-T-MON:/home/ubuntu# cd /tmp
root@RSB-VWA-T-MON:/tmp# du -sk ./* | sort -rn
12 ./snap-private-tmp
8 ./systemd-private-34f7d5063fba4371abc24141bb870b0d-systemd-timesyncd.service-TdzyCZ
8 ./systemd-private-34f7d5063fba4371abc24141bb870b0d-systemd-resolved.service-zF4z18
8 ./systemd-private-34f7d5063fba4371abc24141bb870b0d-systemd-logind.service-9g8JDW
4 ./systemd-private-34f7d5063fba4371abc24141bb870b0d-grafana-server.service-6E90vJ

[tgriep]

I suspect it could be a permission issue for the folders.

Make sure the nagios user can read and write to the 2 folders shown in the errors.


Else it may be that the inodes on the drive is filled up. Rhu the following as root and post the output.

Code: Select all

df -i
df -h
ls -al /
ls -al /usr/local/nagios/var/spool

And, if the /usr/local/nagios/var/spool/checkresults folder is full of files, that could be another cause so delete the files in the /usr/local/nagios/var/spool/checkresults folder.

Thank You.

[ptran]

I suspect it could be a permission issue for the folders.

Make sure the nagios user can read and write to the 2 folders shown in the errors.


Else it may be that the inodes on the drive is filled up. Rhu the following as root and post the output.

Code: Select all

df -i
df -h
ls -al /
ls -al /usr/local/nagios/var/spool

And, if the /usr/local/nagios/var/spool/checkresults folder is full of files, that could be another cause so delete the files in the /usr/local/nagios/var/spool/checkresults folder.

Thank You.

How can I check if the nagios user has read and write access to the 2 mentioned folders?

Hereby the results of the requested commands. There are no files in the checkresults folder.

root@RSB-VWA-T-MON:/home/ubuntu# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
tmpfs 999426 1486 997940 1% /run
/dev/mmcblk0p2 3754080 3754080 0 100% /
tmpfs 999426 1 999425 1% /dev/shm
tmpfs 999426 4 999422 1% /run/lock
/dev/mmcblk0p1 0 0 0 - /boot/firmware
tmpfs 199885 25 199860 1% /run/user/1000
root@RSB-VWA-T-MON:/home/ubuntu# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 781M 20M 762M 3% /run
/dev/mmcblk0p2 59G 46G 10G 83% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/mmcblk0p1 253M 122M 131M 49% /boot/firmware
tmpfs 781M 4.0K 781M 1% /run/user/1000
root@RSB-VWA-T-MON:/home/ubuntu# ls -al /
total 10756
drwxr-xr-x 20 root root 4096 Jan 22 11:12 .
drwxr-xr-x 20 root root 4096 Jan 22 11:12 ..
drwxr-xr-x 2 root root 4096 Jul 31 2020 .disk
lrwxrwxrwx 1 root root 7 Jul 31 2020 bin -> usr/bin
drwxr-xr-x 4 root root 4096 Jan 16 07:01 boot
-rw------- 1 root root 10928128 Oct 11 2022 core
drwxr-xr-x 17 root root 14020 Jan 22 11:12 dev
drwxr-xr-x 122 root root 12288 Jan 16 06:55 etc
drwxr-xr-x 5 root root 4096 Jun 8 2021 home
lrwxrwxrwx 1 root root 7 Jul 31 2020 lib -> usr/lib
drwx------ 2 root root 16384 Jul 31 2020 lost+found
drwxr-xr-x 2 root root 4096 Jul 31 2020 media
drwxr-xr-x 2 root root 4096 Jul 31 2020 mnt
drwxr-xr-x 3 root root 4096 Feb 7 2022 opt
dr-xr-xr-x 656 root root 0 Jan 1 1970 proc
drwx------ 6 root root 4096 May 10 2023 root
drwxr-xr-x 32 root root 980 Jan 22 11:15 run
lrwxrwxrwx 1 root root 8 Jul 31 2020 sbin -> usr/sbin
drwxr-xr-x 7 root root 4096 Sep 15 2021 snap
drwxr-xr-x 3 root root 4096 Oct 27 2020 srv
dr-xr-xr-x 12 root root 0 Jan 1 1970 sys
-rw-r--r-- 1 root root 0 Jun 10 2021 syslog
drwxrwxrwt 10 root root 4096 Jan 22 11:12 tmp
drwxr-xr-x 11 root root 4096 Feb 4 2022 usr
drwxr-xr-x 14 root root 4096 Oct 20 2020 var
root@RSB-VWA-T-MON:/home/ubuntu# ls -al /usr/local/nagios/var/spool
total 169080
drwxr-xr-x 4 nagios nagios 4096 Feb 7 2022 .
drwxr-xr-x 5 nagios nagios 4096 Jan 22 11:12 ..
drwxrwxr-x 2 nagios nagcmd 4096 Jan 22 11:12 checkresults
drwxr-xr-x 2 nagios nagios 173113344 Jan 22 11:12 nagfluxperfdata
root@RSB-VWA-T-MON:/home/ubuntu# cd /usr/local/nagios/var/spool/checkresults
root@RSB-VWA-T-MON:/usr/local/nagios/var/spool/checkresults# ls -ls
total 0

[ptran]

I installed on my windows pc a program to read out the linux partition of the SD card on which nagios is installed. There it indicates that on the 64 GB SD card, there is still 12 GB free diskspace left, so why is Nagios complaining that there is no space left? Can you restrict in Linux that certain folders can grow in size?

[cnorell]

It looks like the

Code: Select all

/dev/mmcblk0p2

directory is out of inodes based on your output posted. Is that the partition Nagios is installed on? If so, I don't know of a way to add inodes without remaking the filesystem.

Best Regards

Cory Norell

[tgriep]

The inodes being full on the drive is the issue. See this when you ran the df -i command.

/dev/mmcblk0p2 3754080 3754080 0 100% /

When the inodes are full, no new files can be created which is what you are having.

Some where on the drive is a folder with a lot of files, (thousands to million of files) and they need to be cleaned up.

Look in the following folders and if you see a lot of files, they can be deleted.

/usr/local/nagios/var/spool/perfdata/
/usr/local/nagios/var/spool/xidpe/
/usr/local/nagios/var/spool/checkresults
/var/spool/snmptt

Thank You.