No space left on device

[kg2857]

sort: cannot create temporary file in '/tmp': No space left on device
ls -l /tmp
What files are there filling the FS?

[ptran]

This is what is in it but I do not have much rights to look further look into the directories (see also attachment) or delete the older folder from last year.

ubuntu@RSB-VWA-T-MON:~$ ls -l /tmp
total 28
drwx------ 3 root root 4096 Jan 15 16:44 snap-private-tmp
drwx------ 3 root root 4096 Jan 15 16:43 systemd-private-4cc7358c0dd14483966d7acddfff2563-ModemManager.service-0yvvA6
drwx------ 3 root root 4096 Jan 16 06:36 systemd-private-4cc7358c0dd14483966d7acddfff2563-apache2.service-cR1ww7
drwx------ 3 root root 4096 Jan 15 16:43 systemd-private-4cc7358c0dd14483966d7acddfff2563-nagios-nrpe-server.service-cCog22
drwx------ 3 root root 4096 Jan 15 16:43 systemd-private-4cc7358c0dd14483966d7acddfff2563-systemd-logind.service-TpkEv2
drwx------ 3 root root 4096 Jan 15 16:42 systemd-private-4cc7358c0dd14483966d7acddfff2563-systemd-resolved.service-4l01K6
drwx------ 3 root root 4096 Mar 20 2023 systemd-private-4cc7358c0dd14483966d7acddfff2563-systemd-timesyncd.service-gfNxj4

[kg2857]

Maybe try searching the internet on those file names.

[ptran]

I found some old log files in the below folder but cannot seem to delete any of them in there. Why is this? I cannot remove them via Filezilla.

/var/log/journal


Response: 550 Delete operation failed.
Command: DELE user-1000@a29954c6204c4030bf82c548b158692d-0000000000052739-0005f9bffc30d021.journal
Response: 550 Delete operation failed.
Command: DELE user-1001@16a29bdf8070402eb23be82be621435b-000000000003c00a-0005f84e9cd443c3.journal
Response: 550 Delete operation failed.
Command: DELE user-1000@a29954c6204c4030bf82c548b158692d-000000000004f26c-0005f9884a548948.journal

[cnorell]

Are you logged in via Filezilla with sufficient permissions to delete the directories? Can you delete the files within those directories, rather than deleting the directory as a whole as a first step?

Perhaps also try SSH'ing directly into the machine rather than using a client like Filezilla. I can't imagine that would change your ability to delete directories, but it's worth a shot if you're able.

Best Regards,

Cory Norell

[kg2857]

I think it's time to go into /tmp as root and use du to find what's using space.

[ptran]

Are you logged in via Filezilla with sufficient permissions to delete the directories? Can you delete the files within those directories, rather than deleting the directory as a whole as a first step?

Perhaps also try SSH'ing directly into the machine rather than using a client like Filezilla. I can't imagine that would change your ability to delete directories, but it's worth a shot if you're able.

Best Regards,

Cory Norell

I have 2 accounts for Nagios but it does not work for both of them to delete these files. I am deleting the files one by one and that does not work. I am not trying to delete the whole folder in one go.

Even when I am connected via putty ssh, it does not work to delete them via this interface.

ubuntu@RSB-VWA-T-MON:/var/log$ cd /var/log/journal/db1a06b71615441ca77b6fa011cac336
ubuntu@RSB-VWA-T-MON:/var/log/journal/db1a06b71615441ca77b6fa011cac336$ rm user-1001@ecc996414e414d64bfdb303170bec0d2-00000000000003d0-00060254d7c268be.journal
rm: remove write-protected regular file 'user-1001@ecc996414e414d64bfdb303170bec0d2-00000000000003d0-00060254d7c268be.journal'? y
rm: cannot remove 'user-1001@ecc996414e414d64bfdb303170bec0d2-00000000000003d0-00060254d7c268be.journal': Permission denied

[ptran]

I think it's time to go into /tmp as root and use du to find what's using space.

This is the result and I am not able to look further into these subfolders to see what resides in them.

ubuntu@RSB-VWA-T-MON:/var/log/journal/db1a06b71615441ca77b6fa011cac336$ du -a /tmp | sort -n -r | head -n 5
du: cannot read directory '/tmp/snap-private-tmp': Permission denied
du: cannot read directory '/tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-resolved.service-569gcx': Permission denied
du: cannot read directory '/tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-timesyncd.service-18Tq09': Permission denied
du: cannot read directory '/tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-ModemManager.service-bHIqPj': Permission denied
du: cannot read directory '/tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-nagios-nrpe-server.service-N1oGd5': Permission denied
du: cannot read directory '/tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-logind.service-Y4h4Gl': Permission denied
48 /tmp
4 /tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-timesyncd.service-18Tq09
4 /tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-resolved.service-569gcx
4 /tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-logind.service-Y4h4Gl
4 /tmp/systemd-private-dae8bdf523554e4e8450ef7743e64508-nagios-nrpe-server.service-N1oGd5

[kg2857]

Are you the root user?
cd /tmp
du -sk ./* | sort -rn

[ptran]

Are you the root user?
cd /tmp
du -sk ./* | sort -rn

I have 2 accounts for the nagios system but I do not know if they are root users? Is there a way to check this?

Both accounts do not seem to have the same access. I do not think they are root user accounts as you cannot delete or view all files and folders with these accounts.

ubuntu@RSB-VWA-T-MON:/var/log/journal/db1a06b71615441ca77b6fa011cac336$ cd /tmp
ubuntu@RSB-VWA-T-MON:/tmp$ du -sk ./* | sort -rn
du: cannot read directory './snap-private-tmp': Permission denied
du: cannot read directory './systemd-private-dae8bdf523554e4e8450ef7743e64508-ModemManager.service-bHIqPj': Permission denied
du: cannot read directory './systemd-private-dae8bdf523554e4e8450ef7743e64508-nagios-nrpe-server.service-N1oGd5': Permission denied
du: cannot read directory './systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-logind.service-Y4h4Gl': Permission denied
du: cannot read directory './systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-resolved.service-569gcx': Permission denied
du: cannot read directory './systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-timesyncd.service-18Tq09': Permission denied
4 ./systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-timesyncd.service-18Tq09
4 ./systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-resolved.service-569gcx
4 ./systemd-private-dae8bdf523554e4e8450ef7743e64508-systemd-logind.service-Y4h4Gl
4 ./systemd-private-dae8bdf523554e4e8450ef7743e64508-nagios-nrpe-server.service-N1oGd5
4 ./systemd-private-dae8bdf523554e4e8450ef7743e64508-ModemManager.service-bHIqPj
4 ./snap-private-tmp