Page 1 of 2
Backups Failing?
Posted: Tue Jun 21, 2016 10:09 am
by CFT6Server
Looks like our backups are failing. The back folder is there and contains files, but looks like it isn't zipping them up. So I suspect that the backups are failing somewhere. Where can I find what went wrong with the backup to remediate?
Code: Select all
ls -l
total 736924
-rw-r--r-- 1 nagios nagios 352277265 Mar 31 12:50 1459453541.tar.gz
-rw-r--r-- 1 nagios nagios 402309799 Apr 6 11:41 1459967722.tar.gz
drwxr-xr-x 4 nagios users 4096 Jun 21 07:38 1466519670
drwxr-xr-x 4 nagios users 4096 Jun 21 08:02 1466521150
Code: Select all
ls 1466521150
conf.d mrtg.tar.gz nagiosql-etc.tar.gz nagios.tar.gz nagvis.tar.gz
mrtg.cfg mysql nagiosql.tar.gz nagiosxi.tar.gz nrdp.tar.gz
Re: Backups Failing?
Posted: Tue Jun 21, 2016 10:16 am
by spurrellian
I've had this issue in the past if there isn't enough space. Have you got plenty available?
Re: Backups Failing?
Posted: Tue Jun 21, 2016 10:23 am
by CFT6Server
Yup, first thing I checked as well since I did run into that before also.
Code: Select all
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
89G 45G 40G 53% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/sda1 477M 66M 386M 15% /boot
Re: Backups Failing?
Posted: Tue Jun 21, 2016 10:47 am
by rkennedy
CFT6Server wrote:Yup, first thing I checked as well since I did run into that before also.
Code: Select all
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
89G 45G 40G 53% /
tmpfs 2.9G 0 2.9G 0% /dev/shm
/dev/sda1 477M 66M 386M 15% /boot
For some background information, are your backups going locally, or to a remote machine? Is the above the output from the local or remote? Nagios will need enough room to zip up any backup archives, which could be causing an issue.
Also, what version of XI are you running?
Re: Backups Failing?
Posted: Tue Jun 21, 2016 10:58 am
by CFT6Server
everything is done locally and we are on 5.2.5
Re: Backups Failing?
Posted: Tue Jun 21, 2016 12:31 pm
by lmiltchev
Remove all of the unzipped folders from the backup directory (unfinished backups). Start a running tail on the cmdsubsys.log:
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
From the Nagios XI web UI, go to Admin->Scheduled Backups->Local, and schedule a backup a few minutes in the future. Examine the "cmdsubsys.log" for clues/errors while it is running.
Hit "Ctrl+c" to exit the tail command when you are done, and post the output on the forum.
Re: Backups Failing?
Posted: Tue Jun 21, 2016 12:43 pm
by CFT6Server
Ah ok.. I see the error now. MySQL pw... however this wasn't changed, perhaps something happened during the upgrade from previous versions?
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
..................................
PROCESSED 0 COMMANDS
COMMAND: CMD=1117, DATA=
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
tar: /usr/local/nagios/share/perfdata/edmbchwiski3: file changed as we read it
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..................................................Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
PROCESSED 0 COMMANDS
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: Got error: 130: Incorrect file format 'nagios_logentries' when using LOCK TABLES
Error backing up MySQL database 'nagios' - check the password in this script!
OUTPUT=Error backing up MySQL database 'nagios' - check the password in this script!
RETURNCODE=2
PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..
PROCESSED 0 COMMANDS
Re: Backups Failing?
Posted: Tue Jun 21, 2016 1:06 pm
by lmiltchev
Log in mysql:
and run:
Code: Select all
repair table nagios_logentries use_frm;
\q
Restart mysqld:
and check the log for errors/crashed tables:
If everything looks good, try running the backup again. Hope this helps.
Re: Backups Failing?
Posted: Tue Jun 21, 2016 4:13 pm
by CFT6Server
This seems to do the trick. I had to repeat the process for a few tables before it completed.
Re: Backups Failing?
Posted: Tue Jun 21, 2016 4:49 pm
by mcapra
Is it ok if we lock this thread and mark the issue as resolved?