mysqld service not running

[alejandro_arroyo]

Dear Support Team,

This morning I received a report indicating that the users cannot login into Nagios XI. After some checks I found that the mysqld service was stopped and I was not able to start it.

The tail of the mysqld.log reads as follows:

[root@nagios log]# tail mysqld.log
190718 23:23:43 InnoDB: Error: unable to create temporary file; errno: 28
190718 23:23:43 [ERROR] Plugin 'InnoDB' init function returned error.
190718 23:23:43 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
190718 23:23:43 [ERROR] Can't start server : Bind on unix socket: No space left on device
190718 23:23:43 [ERROR] Do you already have another mysqld server running on socket: /var/lib/mysql/mysql.sock ?
190718 23:23:43 [ERROR] Aborting
190718 23:23:43 [Note] /usr/libexec/mysqld: Shutdown complete
190718 23:23:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

I do have space on all partitions:

[root@nagios tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 24G 12G 69% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 95M 357M 21% /boot

I also tried to run the repair_databases.sh script without success.

I rebooted the virtual machine and the problem continues.

Found that on /var/tmp are some directories full of files: check_oracle_health oradiag_nagios oradiag_root

the amount of files is so big that I ran the command find /var/tmp -type f | wc -l to count the files and I aborted it after 30 minutes without any response.

These files are related to an oracle db package installed following nagios xi recommendations in order to monitor several oracle instances.

I believe this unusual amount of files is causing the tmp filesystem to collapse and myslqd cannot run in these conditions.

Any ideas son how to solve this issue?

Thanks,

Alejandro Arroyo

[lmiltchev]

Can you run the following command, and show the output?

Code: Select all

df -ih

It is usually safe to delete files from /var/tmp directory. You will have to do this in order to free some space. This folder is usually cleared automatically every 30 days:

Code: Select all

# cat /usr/lib/tmpfiles.d/tmp.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

# See tmpfiles.d(5) for details

# Clear tmp directories separately, to make them easier to override
v /tmp 1777 root root 10d
v /var/tmp 1777 root root 30d

# Exclude namespace mountpoints created with PrivateTmp=yes
x /tmp/systemd-private-%b-*
X /tmp/systemd-private-%b-*/tmp
x /var/tmp/systemd-private-%b-*
X /var/tmp/systemd-private-%b-*/tmp

You can probably adjust the settings in the tmp.conf to clear the folder more frequently.

If you still cannot start mysqld, you can try removing the socket, then starting mysqld.

Code: Select all

rm -f /var/lib/mysql/mysql.sock

Code: Select all

service mysqld start

or

Code: Select all

service mariadb start

[alejandro_arroyo]

After posting yesterday, I delete some files on /tmp and /var/tmp. After that I started mysqld and also ran the repair_databases.sh script.

This is the output you requested:

[root@nagios ~]# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root 2.4M 2.3M 99K 96% /
tmpfs 479K 1 479K 1% /dev/shm
/dev/sda1 126K 56 125K 1% /boot

Inode space is almost full, probably related to the large number of files on /tmp

I'll go ahead and empty /var/tmp directory.

Let you know how it goes...

Alejandro

[lmiltchev]

Let us know if deleting the files from /var/tmp directory solves your problem. Thanks!

[alejandro_arroyo]

Deleting /var/tmp makes all the difference. Take a look:


[root@nagios ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
37G 12G 24G 34% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
/dev/sda1 477M 95M 357M 21% /boot

[root@nagios ~]# df -ih
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
2.4M 116K 2.3M 5% /
tmpfs 479K 1 479K 1% /dev/shm
/dev/sda1 126K 56 125K 1% /boot
[root@nagios ~]#

I wonder if is normal that those Oracle monitoring plugins generate that many files...

Thanks,

Alejandro

[lmiltchev]

Deleting /var/tmp makes all the difference.

I am glad your issue has been resolved!

I wonder if is normal that those Oracle monitoring plugins generate that many files...

It's not unusual. You will need to adjust your settings to clear these files more frequently, so that you are not going to run out of disk space/inodes.

Let us know if it is safe to close this topic now. Thank you!