Backup Nagios XI server - setup

[jacek]

Hi,

I'm aiming at setting up an backup/test Nagios XI server in our environment.
My idea is to:

• Clone the existing VM that the original Nagios XI server is running to an backup server in a different location
• On regular (automatic) basis download a fresh backup (generated by Nagios XI) from the original server and restore it to the cloned machine
• Keep monitoring and notifications off on the cloned machine

I'm hoping that this would give me an almost 1:1 setup so in case something goes wrong with the original server we can start monitoring from the backup (worst case scenario) and also to test some solutions and updates of Nagios XI before deploying them to PROD.

Anyway - my questions are:

• How do I make sure that after the restore from backup Nagios will not re-enable monitoring and notifications
• How do I turn off notifications completely (not sending alone, also generating them, so in case we do some tests and turn notifications on it will not send a bunch of old mails that stacked up over some time)
• Is there a way to redirect all notifications to one email address (debug/test purpose) by modifying some script/settings?

Thanks!

[benjaminsmith]

Hi @Jacek,

How do I make sure that after the restore from backup Nagios will not re-enable monitoring and notifications
How do I turn off notifications completely (not sending alone, also generating them, so in case we do some tests and turn notifications on it will not send a bunch of old mails that stacked up over some time)

I would recommend disabling checks and notifications globally in the main nagios configuration file /usr/local/nagios/etc/nagios.cfg, set the following values to:

Code: Select all

enable_notifications=0
execute_service_checks=0
execute_host_checks=0

This will put Nagios Core into a 'sleep mode' ( see Main Configuration File Options ).

Is there a way to redirect all notifications to one email address (debug/test purpose) by modifying some script/settings?

Probably the best way to do this would be to change the notification handler Nagios uses to send emails to contacts. In the command string there is a macro that will reference the email address of the contacts associated with the host or service.

You can swap this macro with your own that points to the test email address instead of using the contact's email. For example, here is the host notification command for Nagios XI users:

Code: Select all

/usr/bin/php /usr/local/nagiosxi/scripts/handle_nagioscore_notification.php --notification-type=host --contact="$CONTACTNAME$" --contactemail="$CONTACTEMAIL$" --type=$NOTIFICATIONTYPE$ --escalated="$NOTIFICATIONISESCALATED$" --author="$NOTIFICATIONAUTHOR$" --comments="$NOTIFICATIONCOMMENT$" --host="$HOSTNAME$" --hostaddress="$HOSTADDRESS$" --hostalias="$HOSTALIAS$" --hostdisplayname="$HOSTDISPLAYNAME$" --hoststate=$HOSTSTATE$ --hoststateid=$HOSTSTATEID$ --lasthoststate=$LASTHOSTSTATE$ --lasthoststateid=$LASTHOSTSTATEID$ --hoststatetype=$HOSTSTATETYPE$ --currentattempt=$HOSTATTEMPT$ --maxattempts=$MAXHOSTATTEMPTS$ --hosteventid=$HOSTEVENTID$ --hostproblemid=$HOSTPROBLEMID$ --hostoutput="$HOSTOUTPUT$" --longhostoutput="$LONGHOSTOUTPUT$" --datetime="$LONGDATETIME$" --number="$HOSTNOTIFICATIONNUMBER$"

and then change $CONTACTEMAIl$ to a new user defined macro with the test email address. You can make changes to commands in Nagios XI by going to Configure > CCM > Commands.

commands.png

Please see the following guides for more information:
Understanding the User Macro Component
Configuring Core Contacts to Use Nagios XI PHPMailer SMTP Settings (for information about notification handlers)

For your reference, below is a link to our guide for backing up and restoring Nagios XI.
Backing Up And Restoring Your Nagios XI System

Also, you can use a separate XI to server to monitor your production system and notify you in the event the system stops running or becomes unreachable.

[jacek]

Sounds good for a start!
AFAIR the restore will overwrite all files on the server, so it will re-enable monitoring and notifications again.

How should I approach this?
Would a copy of the restore script (/usr/local/nagiosxi/scripts/restore_xi.sh) to a safe place (like some home directory) work? Or does it need to be in the original directory?
I'm thinking about modifying the script to disable the monitoring and notifications just before it starts the nagios service and using the modified script to restore from the backups on a daily basis.
Didn't take a look at the script yet, but hoping this is a good way to achieve that?

I would also need to correct the "Program URL" in "Admin > System Config > System Settings"
Any hint where I can change that from the script, so in files/DB?

[npolovenko]

@jacek, The same script that would restore nagiosxi from a backup can do the following.

Use a shared network storage device to store XI backups from the primary server.

On the backup XI server once a day run a script that would:
1. restore from a backup
2. stop nagios with service nagios stop
3. disable notifications by setting enable_notifications to 0 in the nagios.cfg file.

Code: Select all

sed -i 's/enable_notifications=1/enable_notifications=0/' /usr/local/nagios/etc/nagios.cfg

4. disable host and service check execution:

Code: Select all

sed -i 's/execute_service_checks=1/execute_service_checks=0/' /usr/local/nagios/etc/nagios.cfg
sed -i 's/execute_host_checks=1/enable_notifications=0/' /usr/local/nagios/etc/nagios.cfg

5. Update mysql Internal url and external url(if needed) using the following queries:

Code: Select all

echo "update xi_options set value = 'http://192.168.3.3/nagiosxi/' where name = 'url';" | mysql -uroot -pnagiosxi nagiosxi

echo "update xi_options set value = 'http://192.168.3.3/nagiosxi/' where name = 'external_url';" | mysql -uroot -pnagiosxi nagiosxi

!These queries haven't been tested! I found which tables need to be updated and came up with queries in a few minutes. Please do not run them on the production system. Just on the backup server when needed.

4. start nagios with /etc/init.d/nagios start