Scheduled backups not working

[tony]

We currently have an issue with scheduled backups in Nagios XI.

We have configured scheduled SSH backups and scheduled local backups. These do not run as per the schedule.

We have been monitoring the /usr/local/nagiosxi/var/ cmdsubsys.log at the time the schedule should run and nothing happens.

We have increased the log level for scheduled backups to debug as per the steps detailed in https://support.nagios.com/kb/article.php?id=578.

Below are the entries from /usr/local/nagiosxi/var/components/scheduledbackups.log file for a scheduled backup job that tried to run:

12-06-2017 12:20:01 DEBUG: Running scheduled SSH backup ...
12-06-2017 12:20:01 DEBUG: Sending create local backup command to CmdSubsystem
12-06-2017 12:42:37 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

It is worth noting that we can create a manual backup successfully via the web interface (Admin > Local Backup Archives > Create Backup) and by running the /usr/local/nagiosxi/scripts/backup_xi.sh

[lmiltchev]

Have you made any changes to the system as modifying sudoers, hardening the server, etc.?

I found a similar issue on the forum that was resolved by fixing the sudoers issues:
https://support.nagios.com/forum/viewto ... 16&t=43598

Can you start a running tail on the cmdsubsys.log:

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log 

schedule a ssh backup in the GUI (a few minutes in the future), and wait until you get some output. Exit with "Ctrl+c", copy/paste the output on the forum.

Also, post the entire "/usr/local/nagiosxi/var/components/scheduledbackups.log" and the sudoersfile.

[tony]

I added the following line to the end of this file /usr/local/nagiosxi/html/config.inc.php
$cfg[‘backup_timeout’] = 3600;

Restarted nagios service then httpd service.


Now not sure if the consistently SSH backup is working as seduledbackups.log is sometimes reporting
- ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

seduledbackups.log
12-06-2017 20:00:01 DEBUG: Running scheduled SSH backup ...
12-06-2017 20:00:01 DEBUG: Sending create local backup command to CmdSubsystem
12-06-2017 20:06:01 DEBUG: Running scheduled SSH backup ...
12-06-2017 20:06:01 DEBUG: Sending create local backup command to CmdSubsystem
12-06-2017 21:07:39 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
12-06-2017 22:30:02 DEBUG: Running scheduled SSH backup ...
12-06-2017 22:30:02 DEBUG: Sending create local backup command to CmdSubsystem
12-06-2017 23:37:35 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
12-07-2017 09:30:01 DEBUG: Running scheduled SSH backup ...
12-07-2017 09:30:01 DEBUG: Sending create local backup command to CmdSubsystem

This seduledbackups.log shows SSH Backup Failed for 2 backups from last night 6th December -

-- Scheduled at 20:06 failed at 21:07
-- Scheduled at 22:30 failed at 23:37 on the
-- I scheduled an SSH backup for 9.30 today 7th December

Looking at the remote backup folder - we do have 3 backups of 3 GB

[[email protected]@pr-ita-nagw01 backups]$ ls -lh
total 12G
-rw-r--r--. 1 [email protected] domain [email protected] 3.0G Dec 6 19:45 nagiosxi.1512588901.tar.gz
-rw-r--r--. 1 [email protected] domain [email protected] 3.0G Dec 6 19:56 nagiosxi.1512589622.tar.gz
-rw-r--r--. 1 [email protected] domain [email protected] 3.0G Dec 6 20:16 nagiosxi.1512590761.tar.gz
-rw-r--r--. 1 [email protected] domain [email protected] 3.0G Dec 7 09:40 nagiosxi.1512639001.tar.gz
-rw-r--r--. 1 [email protected] domain [email protected] 110 Dec 6 07:41 ssh_test_secure_copy.txt
[[email protected]@pr-ita-nagw01 backups]$


And the /usr/local/nagiosxi/var/cmdsubsys.log shows backup complete for the 9.30 backup today

===============
BACKUP COMPLETE
===============
Backup stored in /store/backups/nagiosxi/nagiosxi.1512639001.tar.gz
OUTPUT=Backup stored in /store/backups/nagiosxi/nagiosxi.1512639001.tar.gz
RETURNCODE=0

PROCESSED 1 COMMANDS
..................PROCESSING COMMAND ID 36826...
PROCESS COMMAND: CMD=1119, DATA=a:2:{i:0;s:19:"nagiosxi.1512639001";i:1;s:24:"/store/backups/nagiosxi/";}
CMDLINE=rm -rf /store/backups/nagiosxi/nagiosxi.1512639001.tar.gz
OUTPUT=
RETURNCODE=0



So why is the cmdsubsys.log sometimes reporting - ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

Is this a timeout issue - do I need to increase it further?

[lmiltchev]

So why is the cmdsubsys.log sometimes reporting - ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist
Is this a timeout issue - do I need to increase it further?

I could be a timeout issue. Many factors can play a role here - the size of the backup, your hardware resources, the network connection, etc. The backup_timeout value of 3600 in the config.inc.php *should* be sufficient but to be sure, you could check how long it usually takes to run a backup on your system.

Code: Select all

cd /usr/local/nagiosxi/scripts
time ./backup_xi.sh

As far as the ssh transfer time - it would depend on your network connection. You could time a ssh transfer a file of approximately 3GB to see how long it is going to take.

[tony]

The SSH backup scheduled for 22:30 last night failed again:

The scheduledbackups.log shows that it returns an error after only 7 minutes?

12-07-2017 22:30:02 DEBUG: Running scheduled SSH backup ...
12-07-2017 22:30:02 DEBUG: Sending create local backup command to CmdSubsystem
12-07-2017 23:37:34 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

Is there any other logs that I can check rsync or ssh etc...

[tony]

Another scheduled backup scheduled for 9.05 failed - the failure message was reported after 1 hour (which may equate to the time value!)

Previously the backups completed after 10 minutes.

12-08-2017 09:05:01 DEBUG: Running scheduled SSH backup ...
12-08-2017 09:05:01 DEBUG: Sending create local backup command to CmdSubsystem
12-08-2017 10:12:42 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

So I am going to increase the backup timeout to 90 minutes - 5400 seconds

[tony]

Increased timeout to 90 minutes - now having t o wait longer (90 mins) before I am informed that the backup failed

12-08-2017 10:35:02 DEBUG: Running scheduled SSH backup ...
12-08-2017 10:35:02 DEBUG: Sending create local backup command to CmdSubsystem
12-08-2017 12:16:40 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

I do not believe that the issue is a network \ connection issue

What other logs can I check for the SSH file transfer

[lmiltchev]

Did you time your local backup? We need to see how long it usually takes to complete a "local" backup. Run the following command, and show the output after the backup is complete.

Code: Select all

cd /usr/local/nagiosxi/scripts
time ./backup_xi.sh

We need to also see the backups that were created successfully, the timestamps, and the permissions of the directories:

Code: Select all

ls -la /store/backups/nagiosxi
ls -lad /store/backups /store/backups/nagiosxi

How many backups were successfully transferred? Show the existing backups on the remote SSH server:

Code: Select all

ls -la /path/to/the/backup/directory

What are your resource limits set at in the php.ini file? Perhaps you need to increase these values if you have an intermittent timeout issues.

Code: Select all

grep 'max_execution_time\|max_input_time\|memory_limit' /etc/php.ini | grep -v ';'

What other logs can I check for the SSH file transfer

It's possible there are some clues in the apache error log. Upload "/usr/local/nagiosxi/var/components/scheduledbackups.log" and "/etc/sudoers" files, and post 100 lines of the apache error log (in code wraps):

Code: Select all

tail -100 /var/log/httpd/error_log

If we don't find anything useful in the tail output, scheduledbackups.log, and sudoers we may have to escalate the issue to our developers. Thank you!

[tony]

Hi,

Please see below for the output of the following commands:

Code: Select all

cd /usr/local/nagiosxi/scripts
time ./backup_xi.sh
ls -la /store/backups/nagiosxi
ls -lad /store/backups /store/backups/nagiosxi
ls -la /path/to/the/backup/directory
grep 'max_execution_time\|max_input_time\|memory_limit' /etc/php.ini | grep -v ';'

I've attached the scheduledbackups.log file as well and will attach the apache log in a separate post. If you still require the sudoers file I will just need to redact some text and send it to you.

Scheduled Backups Details XI.jpg

Backup Destination Details.JPG

[tony]

Hi,

Please find the apache log file attached.

Regards,
Tony