Local Scheduled Backup failure

[NetNeutron]

# ls -lR /usr/local/nagiosxi/html/includes/components/scheduledbackups/

Code: Select all

/usr/local/nagiosxi/html/includes/components/scheduledbackups/:
total 432
-rw-r----- 1 nagios nagios   5803 Dec 13 09:25 ajaxcalls.php
-rw-r----- 1 nagios nagios   2025 Dec 13 09:25 CHANGELOG.txt
drwxr-x--- 2 nagios nagios   4096 Aug 16 10:13 images
drwxr-x--- 2 nagios nagios   4096 Aug 16 10:13 includes
-rwxr-x--- 1 nagios nagios    323 Dec 13 09:25 install.sh
-rw-r----- 1 nagios nagios  93972 Dec 13 09:25 localbackups.php
-rw-r----- 1 nagios nagios 171636 Dec 13 09:25 managebackups.php
-rw-r----- 1 nagios nagios 149076 Dec 13 09:25 scheduledbackups.inc.php

/usr/local/nagiosxi/html/includes/components/scheduledbackups/images:
total 16
-rw-r----- 1 nagios nagios  781 Dec 13 09:25 accept.png
-rw-r----- 1 nagios nagios 4178 Dec 13 09:25 ajax-loader.gif
-rw-r----- 1 nagios nagios  701 Dec 13 09:25 exclamation.png

/usr/local/nagiosxi/html/includes/components/scheduledbackups/includes:
total 8
-rw-r----- 1 nagios nagios  93 Dec 13 09:25 ftp_test_upload.txt
-rw-r----- 1 nagios nagios 110 Dec 13 09:25 ssh_test_secure_copy.txt

[tgriep]

Thanks for the Log files. Bad thing is that there is nothing bad in it.
You can either move this to a support ticket by emailing [email protected] and linking this post or you can PM me your system profile.

To send us your system profile. login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and email it back to us.

[tgriep]

Thanks for the Profile. I am seeing some issues with MYSQL database and that could cause the issue you are having.
Login as root to the XI server and run the following to repair the database.

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --use_frm --databases nagios nagiosql nagiosxi

Then to be safe, run the following to clear out the message queue of any old data trying to to be written to the database.
Run the following as root.

Code: Select all

service nagios stop
service ndo2db stop
for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
service ndo2db start
service nagios start

Then run this and post the output.

Code: Select all

ipcs -q
mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

Then try and schedule a backup to see if it works.

[NetNeutron]

# ipcs -q

Code: Select all

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
0xaf000002 327680     nagios     600        30541824     29826

# mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

Code: Select all

+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 152   |
+----------------------+-------+
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 151   |
+-----------------+-------+

[tgriep]

The max_connections to the MYSQL server are not high enough and that is causing issues when the Nagios processes are writing to the mysql database.
To increase those settings, edit the /etc/my.cnf file

Under the [mysqld] section add the following setting:

Code: Select all

max_connections = 300

Save the file and restart mysql by running the following.

Code: Select all

service mysql restart

Then re-run the repair of the mysql database by running

Code: Select all

mysqlcheck -f -r -u root -pnagiosxi --use_frm --databases nagios nagiosql nagiosxi

Delete the current scheduled local backup, recreate it and see if that fixes the issue.

[SteveBeauchemin]

Another possibility maybe?

Did sudoers get an update for 5.4 that added this

Code: Select all

NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *

If that is missing would backup fail?

Just a thought.

Steve B

[tgriep]

Thanks SteveBeauchemin for the tip.

[NetNeutron]

Still no joy.

# mysql -u root -pnagiosxi -e "show global status like '%used_connections%'; show variables like 'max_connections';"

Code: Select all

+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 110   |
+----------------------+-------+
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 300   |
+-----------------+-------+

I did the restart and the repair.
This line is in sudoers

Code: Select all

NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *

I did a local backup 10 minutes into the future
# cat /usr/local/nagiosxi/var/components/scheduledbackups.log

Code: Select all

01-18-2017 07:10:02 DEBUG: Running scheduled local backup ...
01-18-2017 07:10:02 INFO: Creating a local backup: nagiosxi.1484741402
01-18-2017 07:10:02 DEBUG: Sending create local backup command to CmdSubsystem

No such file or directory exists in /store/backups/nagiosxi
# ll /store/backups/nagiosxi/ | grep 1484741402

Code: Select all

#

Temporarily I am running backup_xi.sh as a cronjob and it is working...

Code: Select all

15 0 * * * /usr/local/nagiosxi/scripts/backup_xi.sh >/dev/null 2>&1

Please advise...

[tgriep]

Let's check these log files for any errors.
Run this as root on the XI server.

Code: Select all

tail -f /usr/local/nagiosxi/var/e*.log

Then submit another scheduled backup job and post the output of the tail command.

[NetNeutron]

Good Morning.
I sent the tail output in a PM.