mysqld fails to start after reboot

[MichielvM]

Hi,

I was wondering if this sounds familiar to anyone:

We have a number of virtual Nagios Xi servers running on ESX.
I have encountered a few occurances that mysqld fails to start after rebooting the virtual machine.
The solution is to move/copy/delete the file /var/lib/mysql/mysqld.sock and start mysqdl again.
It will then create a new sock file and all is well again.

[slansing]

How are you rebooting the machine? It sounds like you are not giving the server enough time to close out the service / process. We ALWAYS recommend using 'shutdown -h now' as a safe way to shut down XI servers. Let us know if the mysqld.lock still sticks around after rebooting in this way, or if it sticks around after you run:

Code: Select all

service mysqld stop

[manymiles]

I, too, have an issue with mysqld. But, I know it is when the server "crashes".

In my case, I had some power problems causing the machine to lose power outright, preventing a graceful shutdown, which prevented the removal of the mysql.sock file, which prevented mysql from starting on the machine's restart. (In my case, it's a metal server, not a VM).

I have read on the net that the mysql.sock file is not removed during a graceless shutdown, and this is not specifically a Nagios issue. But, I find it to be crazy that a power interruption will require operator intervention to restore mysql, which in turn will restore Nagios.

Does Nagios support have a recommendation on this? Can I adjust the init.d/mysqld script to always remove the socket on start/restart? Do you have a recommended script to check/fix for such a condition?

Also, feel free to shard this into another thread if it is more appropriate.

Miles

[slansing]

You could have it validate that there is no .sock, if there is, remove it, then start. But otherwise yes, if there is still a file lying round such as the .sock then that will need to be removed before starting up the mysql daemon.

[manymiles]

Thanks for your response slansing.

In case anyone else is interested, I've modified my original /etc/init.d/mysqld file as follows.

Original mysqld (starting at line 118)

Code: Select all

         if [ -S "$socketfile" ] ; then
            echo "Another MySQL daemon already running with the same unix socket."
            action $"Starting $prog: " /bin/false
            return 1
         fi

New:

Code: Select all

        if [ -S "$socketfile" ] ; then
            # If a graceless shutdown occurred (power outage, etc), a socket file
            # may still exist even when no mysqld process is running.  Since we
            # previously checked above if mysqld was running, we can safely
            # remove the socket file now.
            rm -f $socketfile
        fi

As I've said above, at least a graceless shutdown won't require operator intervention to (ulitmately) restore Nagios to its full glory.

[sreinhardt]

The only caveat that I would mention, specifically regarding hard or unexpected shutdowns, is the potential for mysql db corruption. Otherwise, nice work!

[MichielvM]

replying to earlier comment: This shutdown was performed by

Code: Select all

shutdown now

I guess -h is more graceful.

Mind you: the Mysql issue doesn't happen on every reboot.
We're in a near complete pre-production state, so reboots are required now and then.

[manymiles]

This shutdown was performed by

Code: Select all

shutdown now

I guess -h is more graceful.

The -h mean halt. I wouldn't think it would matter if you don't have it.

Perhaps shutdown is not properly calling "/etc/init.d/mysqld stop" or "/etc/init.d/mysqld stop" is not completing before the OS reboots?

If that's the case, you might want to try my modded script as a workaround so that even if the stop action doesn't complete, you'll still end up with a started Nagios system on reboot.

[MichielvM]

I agree with your remark that it should be standard procedure for mysql to check for a lock file on startup.
Will take a look at your mod.

It's no big issue when you are aware of this behaviour and the reboot is scheduled, but as we all know this usually happens when no one is looking.
(..or your out at night on a dark country road with a flat tire and then your hit by a rainstorm of biblical proportions - derived from a dutch proverb )

[slansing]

(..or your out at night on a dark country road with a flat tire and then your hit by a rainstorm of biblical proportions - derived from a dutch proverb )

Hahaha, well I'm going to throw this in an internal feature request, though it is an issue with the mysql daemon. I'm not sure we will be changing it's init script and shipping the change.