Backups Failing?

[CFT6Server]

Looks like our backups are failing. The back folder is there and contains files, but looks like it isn't zipping them up. So I suspect that the backups are failing somewhere. Where can I find what went wrong with the backup to remediate?

Code: Select all

ls -l
total 736924
-rw-r--r-- 1 nagios nagios 352277265 Mar 31 12:50 1459453541.tar.gz
-rw-r--r-- 1 nagios nagios 402309799 Apr  6 11:41 1459967722.tar.gz
drwxr-xr-x 4 nagios users       4096 Jun 21 07:38 1466519670
drwxr-xr-x 4 nagios users       4096 Jun 21 08:02 1466521150

Code: Select all

ls 1466521150
conf.d    mrtg.tar.gz  nagiosql-etc.tar.gz  nagios.tar.gz    nagvis.tar.gz
mrtg.cfg  mysql        nagiosql.tar.gz      nagiosxi.tar.gz  nrdp.tar.gz

[spurrellian]

I've had this issue in the past if there isn't enough space. Have you got plenty available?

[CFT6Server]

Yup, first thing I checked as well since I did run into that before also.

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       89G   45G   40G  53% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/sda1             477M   66M  386M  15% /boot

[rkennedy]

Yup, first thing I checked as well since I did run into that before also.

Code: Select all

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
                       89G   45G   40G  53% /
tmpfs                 2.9G     0  2.9G   0% /dev/shm
/dev/sda1             477M   66M  386M  15% /boot

For some background information, are your backups going locally, or to a remote machine? Is the above the output from the local or remote? Nagios will need enough room to zip up any backup archives, which could be causing an issue.

Also, what version of XI are you running?

[CFT6Server]

everything is done locally and we are on 5.2.5

[lmiltchev]

Remove all of the unzipped folders from the backup directory (unfinished backups). Start a running tail on the cmdsubsys.log:

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

From the Nagios XI web UI, go to Admin->Scheduled Backups->Local, and schedule a backup a few minutes in the future. Examine the "cmdsubsys.log" for clues/errors while it is running.

Hit "Ctrl+c" to exit the tail command when you are done, and post the output on the forum.

[CFT6Server]

Ah ok.. I see the error now. MySQL pw... however this wasn't changed, perhaps something happened during the upgrade from previous versions?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
..................................
PROCESSED 0 COMMANDS
 COMMAND: CMD=1117, DATA=
CMDLINE=/usr/local/nagiosxi/scripts/backup_xi.sh
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
............tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
tar: /usr/local/nagios/var: file changed as we read it
tar: /usr/local/nagios/share/perfdata/edmbchwiski3: file changed as we read it
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..................................................Backing up Nagios XI...
tar: Removing leading `/' from member names
Backing up MRTG...
tar: Removing leading `/' from member names
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 0 COMMANDS
Backing up NRDP...
tar: Removing leading `/' from member names
Backing up Nagvis...
tar: Removing leading `/' from member names
Backing up MySQL databases...
mysqldump: Got error: 130: Incorrect file format 'nagios_logentries' when using LOCK TABLES
Error backing up MySQL database 'nagios' - check the password in this script!
OUTPUT=Error backing up MySQL database 'nagios' - check the password in this script!
RETURNCODE=2

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
..
PROCESSED 0 COMMANDS

[lmiltchev]

Log in mysql:

Code: Select all

mysql -u root -p'password' nagios

and run:

Code: Select all

repair table nagios_logentries use_frm;
\q

Restart mysqld:

Code: Select all

service mysqld restart

and check the log for errors/crashed tables:

Code: Select all

tail /var/log/mysqld.log

If everything looks good, try running the backup again. Hope this helps.

[CFT6Server]

This seems to do the trick. I had to repeat the process for a few tables before it completed.

[mcapra]

Is it ok if we lock this thread and mark the issue as resolved?