Veeam Backup with Nagios
Veeam Backup with Nagios
Hi all,
Does anybody use the Veeam Linux Agent to backup their Nagios machines?
Im having issues with mine and in consultation with Veeam support and we have narrowed the issue down to an I/O timeout issue that causes the Veeam snapshot to stall.
With Nagios stopped the backup runs through perfectly.
Veeam have passed the issue back to their R+D team but they have no other documented known instances of the 2 not enjoying each others company!
On a digging mission really to see if anybody uses this backup solution?, did you have any issues with it?, is there anything from nagios staff POV of any changes can be made in Nagios with regards to I/O?
Im running an older version of Nagios granted (3.4.1) on a Ubuntu 12.04 machine so yes a little out dated - but it works (bar this Veeam issue that Ive just introduced)
Thanks
Does anybody use the Veeam Linux Agent to backup their Nagios machines?
Im having issues with mine and in consultation with Veeam support and we have narrowed the issue down to an I/O timeout issue that causes the Veeam snapshot to stall.
With Nagios stopped the backup runs through perfectly.
Veeam have passed the issue back to their R+D team but they have no other documented known instances of the 2 not enjoying each others company!
On a digging mission really to see if anybody uses this backup solution?, did you have any issues with it?, is there anything from nagios staff POV of any changes can be made in Nagios with regards to I/O?
Im running an older version of Nagios granted (3.4.1) on a Ubuntu 12.04 machine so yes a little out dated - but it works (bar this Veeam issue that Ive just introduced)
Thanks
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Veeam Backup with Nagios
What's the output of the following: iostat -x 5 5? (it'll run five times, so just let it finish those five)
- tacolover101
- Posts: 432
- Joined: Mon Apr 10, 2017 11:55 am
Re: Veeam Backup with Nagios
what are you backing up as veeam? nagios is disk intensive with files constantly being written / read through the disk. depending on what you were doing, then yes, you could see issues.
might make more sense to only back up x folders, that are actually relevant to what's running on nagios.
might make more sense to only back up x folders, that are actually relevant to what's running on nagios.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Veeam Backup with Nagios
thanks @tacolover101!
What @tacolover101 says is accurate.
Below is a slightly modified (for more logging) version of our XI backup script. It should give you some ideas as to what is important to backup. Locations may differ in Core, so be sure to double-check.
What @tacolover101 says is accurate.
Below is a slightly modified (for more logging) version of our XI backup script. It should give you some ideas as to what is important to backup. Locations may differ in Core, so be sure to double-check.
Code: Select all
#!/bin/bash
BASEDIR=$(dirname $(readlink -f $0))
#this echo is useful for the file but no time to fix
echo "IMPORT ALL XI CFG VARS" | tee -a $SBLOG
. $BASEDIR/../var/xi-sys.cfg
php $BASEDIR/import_xiconfig.php > $BASEDIR/config.dat
. $BASEDIR/config.dat
rm -rf $BASEDIR/config.dat
SBLOG="/usr/local/nagiosxi/var/components/scheduledbackups.log"
ts=`date +%s`
###############################
# USAGE / HELP
###############################
usage () {
echo ""
echo "Use this script to backup Nagios XI."
echo ""
echo " -n | --name Set the name of the backup minus the .tar.gz"
echo " -p | --prepend Prepend a string to the .tar.gz name"
echo " -a | --append Append a string to the .tar.gz name"
echo " -d | --directory Change the directory to store the compressed backup"
echo ""
}
###############################
echo "ADDING LOGIC FOR NEW BACKUPS" | tee -a $SBLOG
###############################
while [ -n "$1" ]; do
case "$1" in
-h | --help)
usage
exit 0
;;
-n | --name)
fullname=$2
;;
-p | --prepend)
prepend=$2"."
;;
-a | --append)
append="."$2
;;
-d | --directory)
rootdir=$2
;;
esac
shift
done
echo "Restart nagios to forcibly update retention.dat" | tee -a $SBLOG
$BASEDIR/manage_services.sh restart nagios
sleep 10
if [ -z $rootdir ]; then
rootdir="/store/backups/nagiosxi"
fi
echo "Move to root dir to store backups" | tee -a $SBLOG
cd $rootdir
#############################
echo "SET THE NAME & TIME" | tee -a $SBLOG
#############################
#seems like as good a place as any for another time stamp
echo `date` | tee -a $SBLOG
name=$fullname
if [ -z $fullname ]; then
name=$prepend$ts$append
fi
echo "Get current Unix timestamp as name" | tee -a $SBLOG
if [ -z $name ]; then
name=$ts
fi
echo "My working directory" | tee -a $SBLOG
mydir=$rootdir/$name
echo "Make directory for this specific backup" | tee -a $SBLOG
mkdir -p $mydir
##############################
echo "BACKUP DIRS" | tee -a $SBLOG
##############################
echo "Backing up Core Config Manager (NagiosQL)..." | tee -a $SBLOG
#cp -rp /var/www/html/nagiosql $mydir
#cp -rp /etc/nagiosql $mydir/nagiosql-etc
tar czfp $mydir/nagiosql.tar.gz /var/www/html/nagiosql
tar czfp $mydir/nagiosql-etc.tar.gz /etc/nagiosql
echo "Backing up Nagios Core..." | tee -a $SBLOG
#cp -rp /usr/local/nagios $mydir
tar czfp $mydir/nagios.tar.gz /usr/local/nagios
echo "Backing up Nagios XI..." | tee -a $SBLOG
#cp -rp /usr/local/nagiosxi $mydir
tar czfp $mydir/nagiosxi.tar.gz /usr/local/nagiosxi
echo "Backing up MRTG..." | tee -a $SBLOG
#cp -rp /usr/local/nagiosxi $mydir
tar czfp $mydir/mrtg.tar.gz /var/lib/mrtg
cp /etc/mrtg/mrtg.cfg $mydir/
cp -r /etc/mrtg/conf.d $mydir/
echo "Backing up NRDP..." | tee -a $SBLOG
tar czfp $mydir/nrdp.tar.gz /usr/local/nrdp
echo "Backing up Nagvis..." | tee -a $SBLOG
tar czfp $mydir/nagvis.tar.gz /usr/local/nagvis
##############################
echo "BACKUP DATABASES" | tee -a $SBLOG
##############################
echo "Backing up MySQL databases..." | tee -a $SBLOG
mkdir -p $mydir/mysql
mysqldump -h $cfg__db_info__ndoutils__dbserver -u $cfg__db_info__ndoutils__user --password="$cfg__db_info__ndoutils__pwd" --add-drop-database -B $cfg__db_info__ndoutils__db > $mydir/mysql/nagios.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagios' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
mysqldump -h $cfg__db_info__nagiosql__dbserver -u $cfg__db_info__nagiosql__user --password="$cfg__db_info__nagiosql__pwd" --add-drop-database -B $cfg__db_info__nagiosql__db > $mydir/mysql/nagiosql.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagiosql' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
# Only backup PostgresQL if we are still using it
if [ $cfg__db_info__nagiosxi__dbtype == "pgsql" ]; then
echo "Backing up PostgresQL databases..." | tee -a $SBLOG
mkdir -p $mydir/pgsql
if [ -z $cfg__db_info__nagiosxi__dbserver ]; then
cfg__db_info__nagiosxi__dbserver="localhost"
fi
pg_dump -h $cfg__db_info__nagiosxi__dbserver -c -U $cfg__db_info__nagiosxi__user $cfg__db_info__nagiosxi__db > $mydir/pgsql/nagiosxi.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up PostgresQL database 'nagiosxi' !" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
else
mysqldump -h "$cfg__db_info__nagiosxi__dbserver" -u $cfg__db_info__nagiosxi__user --password="$cfg__db_info__nagiosxi__pwd" --add-drop-database -B $cfg__db_info__nagiosxi__db > $mydir/mysql/nagiosxi.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagiosxi' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
fi
##############################
# BACKUP CRONJOB ENTRIES
##############################
# Not necessary
##############################
# BACKUP SUDOERS
##############################
# Not necessary
##############################
# BACKUP LOGROTATE
##############################
echo "Backing up logrotate config files..." | tee -a $SBLOG
mkdir -p $mydir/logrotate
cp -rp /etc/logrotate.d/nagiosxi $mydir/logrotate
##############################
# BACKUP APACHE CONFIG FILES
##############################
echo "Backing up Apache config files..." | tee -a $SBLOG
mkdir -p $mydir/httpd
cp -rp $httpdconfdir/nagios.conf $mydir/httpd
cp -rp $httpdconfdir/nagiosxi.conf $mydir/httpd
cp -rp $httpdconfdir/nagiosql.conf $mydir/httpd
##############################
# COMPRESS BACKUP
##############################
echo "Compressing backup..." | tee -a $SBLOG
tar czfp $name.tar.gz $name
rm -rf $name
echo "change ownership" | tee -a $SBLOG
chown $nagiosuser:$nagiosgroup $name.tar.gz
if [ -s $name.tar.gz ];then
echo " " | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "BACKUP COMPLETE" | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "Backup stored in $rootdir/$name.tar.gz" | tee -a $SBLOG
exit 0;
else
echo " " | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "BACKUP FAILED" | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "File was not created at $rootdir/$name.tar.gz" | tee -a $SBLOG
rm -r $mydir
exit 1;
fi
Re: Veeam Backup with Nagios
Trying to use Veeam to make a system image backup for restore to similar/bare metal server.
Basically a no go then.
I back up all the config files anyway to an offsite location so in case of restore ive got all my configs but its all the little things, and packages installed over the years that would take ages to remember and re-configure in a server crash scenario i thought it would be nice just to take a system image of it and my whole server with all its bells and whistles would be restored in minutes rather than hours.
Basically a no go then.
I back up all the config files anyway to an offsite location so in case of restore ive got all my configs but its all the little things, and packages installed over the years that would take ages to remember and re-configure in a server crash scenario i thought it would be nice just to take a system image of it and my whole server with all its bells and whistles would be restored in minutes rather than hours.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Veeam Backup with Nagios
Did you want to continue to troubleshoot this?FTL wrote: Basically a no go then.
I asked for the output of iostat -x 5 5 during the backup, but we never got that.
Re: Veeam Backup with Nagios
Apologies, when you said the next poster was correct in his post i assumed you agreed that it was a no go
Code: Select all
root@EV-NAGIOS:~# iostat -x 5 5
Linux 3.2.0-32-generic-pae (EV-NAGIOS) 18/04/17 _i686_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.28 0.02 0.77 0.29 0.00 97.64
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.00 11.41 9.95 13.97 45.32 142.29 15.68 0.99 41.42 2.93 68.84 1.29 3.09
avg-cpu: %user %nice %system %iowait %steal %idle
2.25 14.12 11.92 19.03 0.00 52.69
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 831.40 79.40 10032.00 141.20 43584.00 914.40 8.75 47.16 4.63 3.17 108.90 0.09 93.12
avg-cpu: %user %nice %system %iowait %steal %idle
1.57 21.09 15.45 30.69 0.00 31.21
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 2507.20 105.00 13268.20 27.40 63495.20 659.20 9.65 35.58 2.68 2.42 127.18 0.07 96.08
avg-cpu: %user %nice %system %iowait %steal %idle
2.46 7.12 6.50 25.60 0.00 58.32
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1682.00 15.40 3445.40 28.60 20648.80 320.80 12.07 8.86 2.57 2.44 17.65 0.11 36.72
avg-cpu: %user %nice %system %iowait %steal %idle
3.42 6.07 6.63 37.70 0.00 46.17
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 2188.40 39.40 2768.00 40.20 20533.60 479.20 14.97 11.32 4.04 3.24 59.20 0.24 67.84
root@EV-NAGIOS:~# iostat -x 5 5
Linux 3.2.0-32-generic-pae (EV-NAGIOS) 18/04/17 _i686_ (4 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.28 0.02 0.77 0.29 0.00 97.64
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1.04 11.41 10.10 13.97 46.06 142.30 15.65 0.99 41.19 2.93 68.84 1.28 3.09
avg-cpu: %user %nice %system %iowait %steal %idle
1.90 6.11 6.21 24.55 0.00 61.22
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 858.40 23.00 4197.80 16.20 20379.20 297.60 9.81 11.01 2.61 2.32 78.37 0.07 31.60
avg-cpu: %user %nice %system %iowait %steal %idle
2.25 6.24 6.19 33.47 0.00 51.84
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 1517.20 34.20 3693.20 97.00 21347.20 726.40 11.65 12.82 3.38 2.80 25.58 0.16 59.92
avg-cpu: %user %nice %system %iowait %steal %idle
1.75 5.92 6.17 24.59 0.00 61.57
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 30.20 5045.40 33.40 20272.00 393.60 8.14 14.27 2.81 2.72 16.53 0.05 27.84
avg-cpu: %user %nice %system %iowait %steal %idle
1.45 7.18 6.71 25.66 0.00 59.01
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 17.00 37.20 7307.40 56.80 29328.00 433.60 8.08 22.85 3.10 2.87 33.46 0.07 48.80
Last edited by FTL on Tue Apr 18, 2017 11:28 am, edited 1 time in total.
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Veeam Backup with Nagios
Well, there's a couple different points of hedging there, so I still think that's correct.tacolover101 wrote:depending on what you were doing, then yes, you could see issues.
We await output.
Re: Veeam Backup with Nagios
output in previous reply.
Thanks
Thanks
-
- Former Nagios Staff
- Posts: 4583
- Joined: Wed Sep 21, 2016 10:29 am
- Location: NoLo, Minneapolis, MN
- Contact:
Re: Veeam Backup with Nagios
Is this running on an SSD or a traditional drive? That is one way to potentially deal with this.
https://assets.nagios.com/downloads/nag ... giosXI.pdf is written for XI, but the basic principles apply to Core.
Another thing you could do is schedule downtime for all of your hosts/services every so often so you can take a full system backup of the nagios server. Maybe once a month you do that, but take daily backups of your configs and such.
https://assets.nagios.com/downloads/nag ... giosXI.pdf is written for XI, but the basic principles apply to Core.
Another thing you could do is schedule downtime for all of your hosts/services every so often so you can take a full system backup of the nagios server. Maybe once a month you do that, but take daily backups of your configs and such.