Backups get stuck

[globalgiving]

The cron job that runs nightly backups never finishes running. Infact, if I don't check on it every day, there will be a build up of the create_backup.sh script (many of them running at the same time)

Here is the output from a manual run of it:

Code: Select all

Starting Nagios Log Server Backup
---------------------------------
Backing up indexes.nagioslogserver ... kibana-int ... nagioslogserver_log ... Completed.
Waiting for backup. This may take a while.
................................................................

Those dots will just keep going forever.

state.json contains this:

Code: Select all

{"count":5,"states":[{"mode":"export","started":"2014-12-01T15:58:32.296Z","path":"file:///store/backups/nagioslogserver/1417449512/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-03T15:58:37.244Z","path":"file:///store/backups/nagioslogserver/1417622317/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-05T15:58:42.944Z","path":"file:///store/backups/nagioslogserver/1417795122/nagioslogserver.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-05T15:58:43.091Z","path":"file:///store/backups/nagioslogserver/1417795122/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-07T15:58:46.964Z","path":"file:///store/backups/nagioslogserver/1417967926/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"}]}

Which shows me that there are multiple backups going on but never have finished. Most of them are stuck on kibana-int.tar.gz. The other files being dumped are complete in each of the backup directories.

Not sure what to do.. When enough of the backup processes build up on the server, it stops accepting new logs from all our servers. So I have to keep on top of this each day and clean up running processes. However, I am not sure how to fix this properly.

[abrist]

Any errors in /var/log/messages?
What are the permissions on:

Code: Select all

kibana-int.tar.gz

[globalgiving]

So yesterday NLS stopped accepting new logs and there were more backup processes running. I rebooted the server (VM) that NLS is running on and everything went back to normal. Checked on the backup situation this morning and it appears that the backup last night completed properly!

So not sure what was wrong, but a good ol' reboot fixed it. Will make a new post in the future if the problem comes back. Sorry to bother you with something that a reboot fixed.

[slansing]

Very odd, well, as more and more people test it in their specific environments, we will smooth more and more of this out, the whole (newly released thing). Thank you for letting us know! We'll close this one down for now.