FTP Backup not working

[kgugerty]

I read a few other posts on this which I believe helped me locate the issue I am experiencing. Unfortunately, they do not cite resolution that actually works in my case. A thread mentioned an update with a bug fix, but I am running a later version, which I would assume would have the fix. (my current version is 5.6.6. I can't upgrade beyond that quite yet.

A topic from 2014 (https://support.nagios.com/forum/viewto ... 16&t=28079) stated:

"It may be hitting a resource limit if your backups are over 500mb each since our in house tests weren't that big."

That thread closed with the issue just seeming to go away.

I do believe that is my issue though. My backup file sizes are 3+ GB in size. watching the scheudledbackups.log I see the message

Code: Select all

 ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist

20 minutes after the start of the backup job.

the tar.gz file is not complete until about 35 minutes after the start of the backup job.

so, my question is, is there a way to adjust the timing?

[jbrunkow]

First off, the /store/backups/nagiosxi directory must exist for scheduled backups to be ran. It also must be accessible by nagios:nagcmd and apache (who is part of the nagcmd group).

Can you also show me what you see in the scheduled backup log for around that time?

Code: Select all

cat /usr/local/nagiosxi/var/components/scheduledbackups.log

How exactly do you want to adjust the timing? Are you thinking that the FTP server can't find the file because it hasn't finished compressing the files yet? The following document may provide the information we need to determine whether this is the case.
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

You could also run the backup from the command line to test how long it takes to create the backup file.

Code: Select all

/usr/local/nagiosxi/scripts/backup_xi.sh

[kgugerty]

Are you thinking that the FTP server can't find the file because it hasn't finished compressing the files yet?

This is exactly what I am saying.

what lead me to this conclusion:

1 I set an automated backup to kick off at 0600.
2 I set a tail on the log using this command:

Code: Select all

tail -f /usr/local/nagiosxi/var/components/scheduledbackups.log

3 I navigated to the directory /store/backups/nagiosxi
4 shortly after 0600 I ran a command

Code: Select all

ls -la 

and saw the backup directory created.
5 at 0620 this line shows up in the log I am tailing

Code: Select all

ERROR: Scheduled FTP Backup Failed: Local file to transfer does not exist

6 I immediately reran the ls -la command and saw that the tar.gz file had not yet been created.
7 at 0635 the tar.gz file was created.
8 at 0639 the tar.gz file was complete.
9 I ran the FTP command manually and uploaded the file to the server. It was successful.
10 I then deleted the file from the nagios server.

I hope this clears things up a bit.

[jbrunkow]

You could be encountering a bug related to creating backups from version 5.6.12. What version of Nagios are you running? Since there has been a bug fix related to this in the past, you may want to consider updating and trying the backup again.

The command subsystem log may also contain information that will be helpful in diagnosing this. Please run the following tail command to capture relevant logs while performing a test backup.

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

We may be able to pinpoint certain problems by looking at your Nagios profile. You can download the profile by navigating to System Profile under System Config on the left pane > and clicking the Download Profile button on that page.

[kgugerty]

I am running version 5.6.6. I monitor 14000 services on 1700 servers. I have some custom checks that tend to break when Nagios is upgraded. I have to run the new versions through a vetting process before running it on production. I do intend to upgrade in the near future though.

I was watching the cmdsubsys.log during the backups and do not recall seeing anything particularly out of the ordinary. I will check it again and post here afterward.

I am curious how the FTP job is kicked off. Is it timed? or is it waiting for an event/signal?

[jbrunkow]

I suppose that depends on how it was set up. What do you see if you navigate to the Admin section of Nagios XI using the bar on top > click Scheduled Backups under System Backups > and look at the FTP Settings under the FTP tab on that page?

Please refer to the following linked documentation for more information on backing up and restoring Nagios XI.
ref = https://assets.nagios.com/downloads/nag ... ios-XI.pdf

I assume that the underlying utility triggering the backup is cron. However, it's probably much easier to manage the job through the XI interface.
https://en.wikipedia.org/wiki/Cron

[kgugerty]

Please refer to the following linked documentation for more information on backing up and restoring Nagios XI.
ref = https://assets.nagios.com/downloads/nag ... ios-XI.pdf

I reviewed this prior to posting here.

I suppose that depends on how it was set up. What do you see if you navigate to the Admin section of Nagios XI using the bar on top > click Scheduled Backups under System Backups > and look at the FTP Settings under the FTP tab on that page?

This allows you to schedule the backup job as a whole. It does not specify the steps. i.e. create the compressed file, then send it via FTP, then remove the compressed file.

I assume that the underlying utility triggering the backup is cron. However, it's probably much easier to manage the job through the XI interface.
https://en.wikipedia.org/wiki/Cron

I actually don't see a refence in the /etc/cron.d/nagiosxi file that mentions the backup process.

Code: Select all

# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   7 * * * root   /root/scripts/autopostgresqlbackup > /dev/null 2>&1

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/recurring_downtime.php >> /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/snmptt_service_results.php >> /usr/local/nagiosxi/var/snmptt_service_results.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php >> /usr/local/nagiosxi/var/deadpool.log 2>&1

Also, I don't think I am running into the issue that was addressed with Version 5.6.12 as this is referring to Tar creation errors.

5.6.12 - 02/27/2020
===================
- Fixed issue with backups not properly generating due to tar creation errors -JO

It generates the tar file just fine in my environment. The issue is, when the FTP job kicks off, the tar file isn't created YET. It's not created, because the files are still copying to the folder. The tar file is ready about 20 minutes AFTER the FTP Job has already failed.

So... my question still remains. What separates the backup creation from the ftp transfer process? looking at how it performs I would guess that it is timed. (I don't see it in /etc/cron.d/nagiosxi as noted above) As in, the system waits a given amount of time prior to kicking off the FTP job. Unfortunately, this time allotment falls way short in my environment. I am looking for a way to adjust that so I do not have to manually transfer the file every week.

[ssax]

Please attach your /etc/php.ini so we can see what resource limits you have set.

If you go to Admin > Scheduled Backups, do you have both Local and FTP enabled? If so, how far apart are they in time? I would recommend at least an hour apart, they are considered separate backup processes.

Click on the Local tab, if the Location field is anything but this:

Code: Select all

/store/backups/nagiosxi

Try changing it to /store/backups/nagiosxi, clicking Update Settings, and testing again. I believe the FTP/SSH settings still use that location for the backups.

Please edit this file:

Code: Select all

/usr/local/nagiosxi/html/includes/constants.inc.php

At the bottom, change this:

Code: Select all

define('SB_LOGLEVEL', 3);

To this:

Code: Select all

define('SB_LOGLEVEL', 0);

Then run this command as root (and leave it running):

Code: Select all

tail -F /var/log/httpd/error_log /var/log/httpd/ssl_error_log /usr/local/nagiosxi/var/cmdsubsys.log /usr/local/nagiosxi/var/components/scheduledbackups.log

Then force a FTP scheduled backup to run and after it's completed or failed, send me the entire output from the tail command above and send me this file:

Code: Select all

/usr/local/nagiosxi/var/components/scheduledbackups.log

Please send the output of these commands (as root):
- NOTE: You may need to adjust the -h 127.0.0.1, the -uroot, and -pnagiosxi in the first command if your DB is offloaded to another server and/or you've changed the root mysql password

Code: Select all

echo "SELECT table_name AS 'Table', round(((data_length + index_length) / 1024 / 1024), 2) 'Size in MB' FROM information_schema.TABLES WHERE table_schema IN ('nagios', 'nagiosql', 'nagiosxi');" | mysql -h 127.0.0.1 -uroot -pnagiosxi --table

Then PM me a copy of your profile if you're able to generate it, you can download it from Admin > System Profile > Download Profile.

That should give us a bit more insight.

Thank you!

[kgugerty]

If you go to Admin > Scheduled Backups, do you have both Local and FTP enabled? If so, how far apart are they in time? I would recommend at least an hour apart, they are considered separate backup processes.

Just FTP Backups enabled.

Try changing it to /store/backups/nagiosxi, clicking Update Settings, and testing again. I believe the FTP/SSH settings still use that location for the backups.

This is the location for the backups.


Log level adjusted and the files requested sent via pm. Please let me know what you see.

[ssax]

It certainly looks like a timing issue from the logs but I haven't seen a timing issue before.

Please create a ticket for this and include a link back to this forum thread so we can get a remote session setup:

https://support.nagios.com/tickets/