Veeam Backup with Nagios

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
FTL
Posts: 72
Joined: Fri Oct 21, 2011 7:23 am

Veeam Backup with Nagios

Post by FTL »

Hi all,

Does anybody use the Veeam Linux Agent to backup their Nagios machines?

Im having issues with mine and in consultation with Veeam support and we have narrowed the issue down to an I/O timeout issue that causes the Veeam snapshot to stall.

With Nagios stopped the backup runs through perfectly.

Veeam have passed the issue back to their R+D team but they have no other documented known instances of the 2 not enjoying each others company!

On a digging mission really to see if anybody uses this backup solution?, did you have any issues with it?, is there anything from nagios staff POV of any changes can be made in Nagios with regards to I/O?

Im running an older version of Nagios granted (3.4.1) on a Ubuntu 12.04 machine so yes a little out dated - but it works (bar this Veeam issue that Ive just introduced)

Thanks
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Veeam Backup with Nagios

Post by dwhitfield »

What's the output of the following: iostat -x 5 5? (it'll run five times, so just let it finish those five)
User avatar
tacolover101
Posts: 432
Joined: Mon Apr 10, 2017 11:55 am

Re: Veeam Backup with Nagios

Post by tacolover101 »

what are you backing up as veeam? nagios is disk intensive with files constantly being written / read through the disk. depending on what you were doing, then yes, you could see issues.

might make more sense to only back up x folders, that are actually relevant to what's running on nagios.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Veeam Backup with Nagios

Post by dwhitfield »

thanks @tacolover101!

What @tacolover101 says is accurate.

Below is a slightly modified (for more logging) version of our XI backup script. It should give you some ideas as to what is important to backup. Locations may differ in Core, so be sure to double-check.

Code: Select all

#!/bin/bash
BASEDIR=$(dirname $(readlink -f $0))
#this echo is useful for the file but no time to fix
echo "IMPORT ALL XI CFG VARS" | tee -a $SBLOG
. $BASEDIR/../var/xi-sys.cfg
php $BASEDIR/import_xiconfig.php > $BASEDIR/config.dat
. $BASEDIR/config.dat
rm -rf $BASEDIR/config.dat
SBLOG="/usr/local/nagiosxi/var/components/scheduledbackups.log"
ts=`date +%s`



###############################
# USAGE / HELP
###############################
usage () {
echo ""
echo "Use this script to backup Nagios XI."
echo ""
echo " -n | --name              Set the name of the backup minus the .tar.gz"
        echo " -p | --prepend           Prepend a string to the .tar.gz name"
        echo " -a | --append            Append a string to the .tar.gz name"
        echo " -d | --directory         Change the directory to store the compressed backup"
echo ""
}

###############################
echo "ADDING LOGIC FOR NEW BACKUPS" | tee -a $SBLOG
###############################
while [ -n "$1" ]; do
case "$1" in
-h | --help)
usage
exit 0
;;
-n | --name)
fullname=$2
;;
-p | --prepend)
prepend=$2"."
;;
-a | --append)
append="."$2
;;
-d | --directory)
rootdir=$2
;;
esac
shift
done

echo "Restart nagios to forcibly update retention.dat" | tee -a $SBLOG
$BASEDIR/manage_services.sh restart nagios
sleep 10

if [ -z $rootdir ]; then
rootdir="/store/backups/nagiosxi"
fi

echo "Move to root dir to store backups" | tee -a $SBLOG
cd $rootdir

#############################
echo "SET THE NAME & TIME" | tee -a $SBLOG
#############################
#seems like as good a place as any for another time stamp
echo `date` | tee -a $SBLOG

name=$fullname

if [ -z $fullname ]; then
name=$prepend$ts$append
fi

echo "Get current Unix timestamp as name"  | tee -a $SBLOG
if [ -z $name ]; then
name=$ts
fi

echo "My working directory" | tee -a $SBLOG
mydir=$rootdir/$name

echo "Make directory for this specific backup"  | tee -a $SBLOG
mkdir -p $mydir

##############################
echo "BACKUP DIRS"  | tee -a $SBLOG
##############################

echo "Backing up Core Config Manager (NagiosQL)..." | tee -a $SBLOG
#cp -rp /var/www/html/nagiosql $mydir
#cp -rp /etc/nagiosql $mydir/nagiosql-etc
tar czfp $mydir/nagiosql.tar.gz /var/www/html/nagiosql
tar czfp $mydir/nagiosql-etc.tar.gz /etc/nagiosql

echo "Backing up Nagios Core..." | tee -a $SBLOG
#cp -rp /usr/local/nagios $mydir
tar czfp $mydir/nagios.tar.gz /usr/local/nagios

echo "Backing up Nagios XI..." | tee -a $SBLOG
#cp -rp /usr/local/nagiosxi $mydir
tar czfp $mydir/nagiosxi.tar.gz /usr/local/nagiosxi

echo "Backing up MRTG..." | tee -a $SBLOG
#cp -rp /usr/local/nagiosxi $mydir
tar czfp $mydir/mrtg.tar.gz /var/lib/mrtg
cp /etc/mrtg/mrtg.cfg $mydir/
cp -r /etc/mrtg/conf.d $mydir/

echo "Backing up NRDP..." | tee -a $SBLOG
tar czfp $mydir/nrdp.tar.gz /usr/local/nrdp

echo "Backing up Nagvis..."  | tee -a $SBLOG
tar czfp $mydir/nagvis.tar.gz /usr/local/nagvis

##############################
echo "BACKUP DATABASES"  | tee -a $SBLOG
##############################
echo "Backing up MySQL databases..." | tee -a $SBLOG
mkdir -p $mydir/mysql
mysqldump -h $cfg__db_info__ndoutils__dbserver -u $cfg__db_info__ndoutils__user --password="$cfg__db_info__ndoutils__pwd" --add-drop-database -B $cfg__db_info__ndoutils__db > $mydir/mysql/nagios.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagios' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
mysqldump -h $cfg__db_info__nagiosql__dbserver -u $cfg__db_info__nagiosql__user --password="$cfg__db_info__nagiosql__pwd" --add-drop-database -B $cfg__db_info__nagiosql__db > $mydir/mysql/nagiosql.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagiosql' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi

# Only backup PostgresQL if we are still using it
if [ $cfg__db_info__nagiosxi__dbtype == "pgsql" ]; then
echo "Backing up PostgresQL databases..." | tee -a $SBLOG
mkdir -p $mydir/pgsql
if [ -z $cfg__db_info__nagiosxi__dbserver ]; then
cfg__db_info__nagiosxi__dbserver="localhost"
fi
pg_dump -h $cfg__db_info__nagiosxi__dbserver -c -U $cfg__db_info__nagiosxi__user $cfg__db_info__nagiosxi__db > $mydir/pgsql/nagiosxi.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up PostgresQL database 'nagiosxi' !" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
else
mysqldump -h "$cfg__db_info__nagiosxi__dbserver" -u $cfg__db_info__nagiosxi__user --password="$cfg__db_info__nagiosxi__pwd" --add-drop-database -B $cfg__db_info__nagiosxi__db > $mydir/mysql/nagiosxi.sql
res=$?
if [ $res != 0 ]; then
echo "Error backing up MySQL database 'nagiosxi' - check the password in this script!" | tee -a $SBLOG
rm -r $mydir
exit $res;
fi
fi

##############################
# BACKUP CRONJOB ENTRIES
##############################
# Not necessary

##############################
# BACKUP SUDOERS
##############################
# Not necessary

##############################
# BACKUP LOGROTATE
##############################
echo "Backing up logrotate config files..." | tee -a $SBLOG
mkdir -p $mydir/logrotate
cp -rp /etc/logrotate.d/nagiosxi $mydir/logrotate

##############################
# BACKUP APACHE CONFIG FILES
##############################
echo "Backing up Apache config files..." | tee -a $SBLOG
mkdir -p $mydir/httpd
cp -rp $httpdconfdir/nagios.conf $mydir/httpd
cp -rp $httpdconfdir/nagiosxi.conf $mydir/httpd
cp -rp $httpdconfdir/nagiosql.conf $mydir/httpd

##############################
# COMPRESS BACKUP
##############################
echo "Compressing backup..." | tee -a $SBLOG
tar czfp $name.tar.gz $name
rm -rf $name

echo "change ownership"  | tee -a $SBLOG
chown $nagiosuser:$nagiosgroup $name.tar.gz

if [ -s $name.tar.gz ];then

echo " " | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "BACKUP COMPLETE" | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "Backup stored in $rootdir/$name.tar.gz" | tee -a $SBLOG

exit 0;
else
echo " " | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "BACKUP FAILED" | tee -a $SBLOG
echo "===============" | tee -a $SBLOG
echo "File was not created at $rootdir/$name.tar.gz" | tee -a $SBLOG
rm -r $mydir
exit 1;
fi

FTL
Posts: 72
Joined: Fri Oct 21, 2011 7:23 am

Re: Veeam Backup with Nagios

Post by FTL »

Trying to use Veeam to make a system image backup for restore to similar/bare metal server.

Basically a no go then.

I back up all the config files anyway to an offsite location so in case of restore ive got all my configs but its all the little things, and packages installed over the years that would take ages to remember and re-configure in a server crash scenario i thought it would be nice just to take a system image of it and my whole server with all its bells and whistles would be restored in minutes rather than hours.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Veeam Backup with Nagios

Post by dwhitfield »

FTL wrote: Basically a no go then.
Did you want to continue to troubleshoot this?

I asked for the output of iostat -x 5 5 during the backup, but we never got that.
FTL
Posts: 72
Joined: Fri Oct 21, 2011 7:23 am

Re: Veeam Backup with Nagios

Post by FTL »

Apologies, when you said the next poster was correct in his post i assumed you agreed that it was a no go

Code: Select all

root@EV-NAGIOS:~# iostat -x 5 5
Linux 3.2.0-32-generic-pae (EV-NAGIOS)     18/04/17     _i686_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.28    0.02    0.77    0.29    0.00   97.64

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               1.00    11.41    9.95   13.97    45.32   142.29    15.68     0.99   41.42    2.93   68.84   1.29   3.09

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.25   14.12   11.92   19.03    0.00   52.69

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda             831.40    79.40 10032.00  141.20 43584.00   914.40     8.75    47.16    4.63    3.17  108.90   0.09  93.12

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.57   21.09   15.45   30.69    0.00   31.21

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda            2507.20   105.00 13268.20   27.40 63495.20   659.20     9.65    35.58    2.68    2.42  127.18   0.07  96.08

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.46    7.12    6.50   25.60    0.00   58.32

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda            1682.00    15.40 3445.40   28.60 20648.80   320.80    12.07     8.86    2.57    2.44   17.65   0.11  36.72

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.42    6.07    6.63   37.70    0.00   46.17

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda            2188.40    39.40 2768.00   40.20 20533.60   479.20    14.97    11.32    4.04    3.24   59.20   0.24  67.84

root@EV-NAGIOS:~# iostat -x 5 5
Linux 3.2.0-32-generic-pae (EV-NAGIOS)     18/04/17     _i686_    (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.28    0.02    0.77    0.29    0.00   97.64

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               1.04    11.41   10.10   13.97    46.06   142.30    15.65     0.99   41.19    2.93   68.84   1.28   3.09

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.90    6.11    6.21   24.55    0.00   61.22

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda             858.40    23.00 4197.80   16.20 20379.20   297.60     9.81    11.01    2.61    2.32   78.37   0.07  31.60

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.25    6.24    6.19   33.47    0.00   51.84

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda            1517.20    34.20 3693.20   97.00 21347.20   726.40    11.65    12.82    3.38    2.80   25.58   0.16  59.92

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.75    5.92    6.17   24.59    0.00   61.57

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    30.20 5045.40   33.40 20272.00   393.60     8.14    14.27    2.81    2.72   16.53   0.05  27.84

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.45    7.18    6.71   25.66    0.00   59.01

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda              17.00    37.20 7307.40   56.80 29328.00   433.60     8.08    22.85    3.10    2.87   33.46   0.07  48.80
Last edited by FTL on Tue Apr 18, 2017 11:28 am, edited 1 time in total.
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Veeam Backup with Nagios

Post by dwhitfield »

tacolover101 wrote:depending on what you were doing, then yes, you could see issues.
Well, there's a couple different points of hedging there, so I still think that's correct.

We await output.
FTL
Posts: 72
Joined: Fri Oct 21, 2011 7:23 am

Re: Veeam Backup with Nagios

Post by FTL »

output in previous reply.

Thanks
dwhitfield
Former Nagios Staff
Posts: 4583
Joined: Wed Sep 21, 2016 10:29 am
Location: NoLo, Minneapolis, MN
Contact:

Re: Veeam Backup with Nagios

Post by dwhitfield »

Is this running on an SSD or a traditional drive? That is one way to potentially deal with this.

https://assets.nagios.com/downloads/nag ... giosXI.pdf is written for XI, but the basic principles apply to Core.

Another thing you could do is schedule downtime for all of your hosts/services every so often so you can take a full system backup of the nagios server. Maybe once a month you do that, but take daily backups of your configs and such.
Locked