Backups get stuck
Posted: Wed Dec 10, 2014 4:18 pm
The cron job that runs nightly backups never finishes running. Infact, if I don't check on it every day, there will be a build up of the create_backup.sh script (many of them running at the same time)
Here is the output from a manual run of it:
Those dots will just keep going forever.
state.json contains this:
Which shows me that there are multiple backups going on but never have finished. Most of them are stuck on kibana-int.tar.gz. The other files being dumped are complete in each of the backup directories.
Not sure what to do.. When enough of the backup processes build up on the server, it stops accepting new logs from all our servers. So I have to keep on top of this each day and clean up running processes. However, I am not sure how to fix this properly.
Here is the output from a manual run of it:
Code: Select all
Starting Nagios Log Server Backup
---------------------------------
Backing up indexes.nagioslogserver ... kibana-int ... nagioslogserver_log ... Completed.
Waiting for backup. This may take a while.
................................................................
state.json contains this:
Code: Select all
{"count":5,"states":[{"mode":"export","started":"2014-12-01T15:58:32.296Z","path":"file:///store/backups/nagioslogserver/1417449512/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-03T15:58:37.244Z","path":"file:///store/backups/nagioslogserver/1417622317/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-05T15:58:42.944Z","path":"file:///store/backups/nagioslogserver/1417795122/nagioslogserver.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-05T15:58:43.091Z","path":"file:///store/backups/nagioslogserver/1417795122/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"},{"mode":"export","started":"2014-12-07T15:58:46.964Z","path":"file:///store/backups/nagioslogserver/1417967926/kibana-int.tar.gz","node_name":"3cb77924-3178-4ba3-8952-1f329efab29c"}]}
Not sure what to do.. When enough of the backup processes build up on the server, it stops accepting new logs from all our servers. So I have to keep on top of this each day and clean up running processes. However, I am not sure how to fix this properly.