Scheduled FTP/SCP Backups Not Working - Part II

[luczynj]

Hello Support,

I'm back with some updated information regarding my previous post titled, "Scheduled FTP/SCP Backups Not Working" post:
https://support.nagios.com/forum/viewto ... 16&t=62891

Here's what I've found so far.

NOTE: Any log output below has been trimmed to only October dates. Also, we are running version 5.7.2.

We have a cronjob for backups:
30 18 * * * /usr/local/nagiosxi/scripts/backup_xi.sh

I have scheduled FTP backups configured to run at 00:48 within the XI GUI.

Here's the output of the files created by the cronjob and the scheduled FTP backups:

[root@nagios-p /store/backups/nagiosxi]# ls -lsart
2891300 -rw-r--r-- 1 nagios nagios 2960683518 Oct 5 01:38 nagiosxi.1633394881.tar.gz <--- scheduled
2901124 -rw-r--r-- 1 nagios nagios 2970746718 Oct 5 19:20 1633458601.tar.gz <--- cron
2902400 -rw-r--r-- 1 nagios nagios 2972050101 Oct 6 01:38 nagiosxi.1633481281.tar.gz <--- scheduled
2911328 -rw-r--r-- 1 nagios nagios 2981194696 Oct 6 19:21 1633545001.tar.gz <--- cron
2912984 -rw-r--r-- 1 nagios nagios 2982890115 Oct 7 01:39 nagiosxi.1633567681.tar.gz <--- scheduled
2916880 -rw-r--r-- 1 nagios nagios 2986878189 Oct 7 19:21 1633631401.tar.gz <--- cron
2918408 -rw-r--r-- 1 nagios nagios 2988443267 Oct 8 01:39 nagiosxi.1633654081.tar.gz <--- scheduled

In the scheduledbackups.log:
[root@nagios-p ~]# tail /usr/local/nagiosxi/var/components/scheduledbackups.log
10-01-2021 01:08:02 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-02-2021 01:08:03 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-03-2021 01:08:04 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-04-2021 01:08:02 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-05-2021 01:08:02 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-06-2021 01:08:03 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-07-2021 01:08:03 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist
10-08-2021 01:08:02 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist


[root@nagios-p nagiosxi]# grep BACKUP /usr/local/nagiosxi/var/components/auditlog.log
2021-10-01 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-01 01:08:02 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-02 00:48:02 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-02 01:08:03 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-03 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-03 01:08:04 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-04 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-04 01:08:02 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-05 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-05 01:08:05 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-06 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-06 01:08:03 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-07 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-07 01:08:03 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys
2021-10-08 00:48:01 - User Interface [32] system:localhost - Submitted COMMAND_CREATE_SYSTEM_BACKUP (ID=1117) to cmdsubsys
2021-10-08 01:08:04 - User Interface [32] system:localhost - Submitted COMMAND_DELETE_SYSTEM_BACKUP (ID=1119) to cmdsubsys


Based on the above, It shows that exactly 20 minutes after the scheduled backup starts, the auditlog reports that it sent "COMMAND_DELETE_SYSTEM_BACKUP" to cmdsubsys, and the scheduledbackups.log file says the file doesn't exist. This is because it is still being built and is incomplete.

How do we get the backup system to increase that 20 minute timeout?

Thank you for your support,
JLu

Nagios XI version 5.7.2
Linux nagios-p 2.6.32-754.33.1.el6.x86_64 #1 SMP Tue Aug 25 15:29:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[tgriep]

At the bottom of this file

Code: Select all

/usr/local/nagiosxi/html/config.inc.php

put the following.

Code: Select all

$cfg['backup_timeout'] = 3600;

Save the change and restart Apache and that will increase the timeout to 60 minutes for the backups.

[luczynj]

Hello,

Thanks for that. I've changed the setting, restarted httpd and am now getting this:

10-10-2021 01:00:01 DEBUG: Sending create local backup command to CmdSubsystem
10-10-2021 02:00:04 ERROR: Scheduled SSH Backup Failed: Local file to transfer does not exist

Any other ideas?

Regards,
JLu

[tgriep]

There might be an error that is causing the backup from finishing so lets run it on a SSH session to see if that is true.

Open a ssh session and run this.

Code: Select all

/usr/local/nagiosxi/scripts/backup_xi.sh

If you see any errors, post them here.

Also, you can enable debugging for the backup to see what the error is when it runs the next time.
See this KB article to enable debugging.
https://support.nagios.com/kb/article/n ... l-578.html

[luczynj]

Sorry for the delayed response.

The issue is still occuring. Can you tell me where, specifically I should put the timeout statement?

You say to put it at the bottom of the file. Here's where I have put it.


###########################################

$cfg['htaccess_file']="/usr/local/nagiosxi/etc/htpasswd.users";
$cfg['htpasswd_path']="/usr/bin/htpasswd";

//
// Added 20211008 as per Nagios Support Forum Question
// regarding timeout of Nagios XI backups.
// https://support.nagios.com/forum/viewto ... 16&t=63704
//

$cfg['backup_timeout']=1500;

$cfg['enable_analytics']=1;


// Secure notification %responseurl%
// setting this to 1 will force users to login in the response URL
// $cfg['secure_response_url']=1;

// Globally disable in page help system
// $cfg['disable_helpsystem']=1;

///////// keep these in order /////////

// include generic db defs
if (!defined('CFG_ONLY')) { if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__).'/includes/db.inc.php'); } }

// include generic definitions
if (!defined('CFG_ONLY')) { if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__).'/db/common.inc.php'); } }
// include db-specific definitions
//if (!defined('CFG_ONLY')) { if (!defined('CFG_ONLY')) { require_once(dirname(__FILE__).'/db/'.$cfg['dbtype'].'.inc.php'); } }

?>

###########################################

Should it go after the last line? The ?>

Thanks

[tgriep]

The location where you have it in the file is OK so you can just update it.

Most systems do not have that option so I just tell everyone to add it to the bottom.

Thank you.

[luczynj]

I've updated the file and the backups appear to be working. However, when I watch the processes during the backup and transfer, the nagios parent ID occasionally spawns a defunct process. I have another script that checks for these processes and if it doesn't = 2, it restarts Nagios.

Any ideas on why the backup would do that?

Regards,
JLu

[tgriep]

When a scheduled backup runs, it restarts the Nagios process to flush the files to the disk so it can get an up to date backup and if there is a check that takes a long time to stop, it could be creating the defunct process.

What time of the day are you running the backup?

Can you get the following files showing the time the backup is running and post them here or PM them to me?

Code: Select all

/usr/local/nagiosxi/var/components/scheduledbackups.log
/usr/local/nagiosxi/var/cmdsubsys.log
/usr/local/nagios/var/nagios.log

Thank you.