Moving server with offloaded DB

[BanditBBS]

Ok, so I want to move my 64bit RHEL 5.x server to a new 64bitRHEL 6.x server. I would normally just use the backup and restore XI, but that includes the DB and well, I don't need to backup and resstore the DB. What options do I have? Would it be as simple as modifying the back and restore scripts and comment out the mysql sections? Do you see any issue with that?

[abrist]

Nope, that will work just fine. Obviously, unless you performed the undocumented postgres offload, leave that portion intact. Let us know if you need assistance doing so.

[BanditBBS]

I finally got my new RHEL 6.x server built. I'm going to follow these steps I think...tell me if I am missing something.....

On New server(RHEL 6.x 64bit)
Make OS changes(ulimit and such)
Install NagiosXI on new server
Configure ramdisk same as old server
Install Oracle Instant client on new server
Modify restore script to skip mysql section
Take VM snapshot!!!!!!!!!!!

On Old server(RHEL 5.x 64bit)
Modify back script to skip mysql section

Actual moving of data:
Shutdown services on old server
Shutdown services on new server
Run backup
Disable network on old server
Run restore
Globally disable notifications on new server
Assign old server IP to new server
Make sure ndo2db is working(its offloaded too)
Reboot new server
Sanity check that everything seems to be working
Enable notifications

My last question is...If I play around in the new sevrer XI for a few(testing a few things before doing this) does the restore overwrite the postgres db, right? The restore will copy over all the config changes I made that made ramdisk, offloaded DB and offloaded ndo2db, right?

SOOOO nervous, lol

[tmcdonald]

My last question is...If I play around in the new sevrer XI for a few(testing a few things before doing this) does the restore overwrite the postgres db, right? The restore will copy over all the config changes I made that made ramdisk, offloaded DB and offloaded ndo2db, right?

This is correct - the backup/restore process includes postgres, as well as the nagios.cfg and other configs that would impact the ramdisk and offloaded DBs.

Your process looks pretty good, just have VM snapshots and an appropriate amount of whiskey close at hand.

[BanditBBS]

Ok, so I am doing my move on Sunday(yes Easter...hey, I live alone, so wtf )

Here is my detailed plan with IPs redacted...does this look right?

Code: Select all

•	Shutdown services on old server
  o	service nagios stop
  o	service npcd stop
•	Shutdown services on new server
  o	service nagios stop
  o	service npcd stop
  o	service ndo2db stop
  o	service mysqld stop
•	Shutdown services on offloaded DB and NDO2DB server
  o	service ndo2db stop
•	On old server Run backup_xi.sh-
  o	cd /usr/local/nagiosxi/scripts/
  o	./backup_xi.sh –n migratenag –d /store/nagios
•	Copy backup to new server-
  o	scp /var/lib/mysql/migratenag* 10.x.x.19:/store
•	Disable network on old server in vcenter
•	On new server run restore_xi.sh-
  o	cd /usr/local/nagiosxi/scripts/
  o	./restore_xi.sh /store/migratenag
•	Globally disable notifications on new server
  o	Go to admin page and click the button
•	Assign old server IP to new server
  o	vi /etc/sysconfig/network-scripts/ifcfg-eth0
    	change 10.x.x.19 to 10.x.x.18
  o	validate correct IP in /etc/hosts
  o	vi /etc/sysconfig/network-scripts/ifcfg-eth1
    	BOOTPROTO=static
    	ONBOOT=yes
    	IPADDR=10.x.x.210
    	NETMASK=255.255.0.0
•	Start services on offloaded DB and NDO2DB server
  o	service ndo2db start
•	Reboot new server
•	Sanity check that everything seems to be working
•	Globally enable notifications on new server
  o	Go to admin page and click the button
•	Set ndo2db to not start on new server
  o	chkconfig ndo2db off

My back out plan in case things go wrong or I forgot to install some things on the new server and too much isn't working is to just shut off the new server and then enable network on the old server again. Should work, right?

Thanks!

[jdalrymple]

Code: Select all

    •   Shutdown services on old server
      o   service nagios stop
      o   service npcd stop
    •   Shutdown services on new server
      o   service nagios stop
      o   service npcd stop
      o   service ndo2db stop
      o   service mysqld stop
    •   Shutdown services on offloaded DB and NDO2DB server
      o   service ndo2db stop
    •   On old server Run backup_xi.sh-
      o   cd /usr/local/nagiosxi/scripts/
      o   ./backup_xi.sh –n migratenag –d /store/nagios
    •   Copy backup to new server-
      o   scp /var/lib/mysql/migratenag* 10.x.x.19:/store
    •   Disable network on old server in vcenter
    •   On new server run restore_xi.sh-
      o   cd /usr/local/nagiosxi/scripts/
      o   ./restore_xi.sh /store/migratenag
    •   Globally disable notifications on new server
      o   Go to admin page and click the button
    •   Assign old server IP to new server
      o   vi /etc/sysconfig/network-scripts/ifcfg-eth0
           change 10.x.x.19 to 10.x.x.18
      o   validate correct IP in /etc/hosts
      o   vi /etc/sysconfig/network-scripts/ifcfg-eth1
           BOOTPROTO=static
           ONBOOT=yes
           IPADDR=10.x.x.210
           NETMASK=255.255.0.0
    •   Start services on offloaded DB and NDO2DB server
      o   service ndo2db start
    •   Reboot new server
    •   Sanity check that everything seems to be working
    •   Globally enable notifications on new server
      o   Go to admin page and click the button
    •   Set ndo2db to not start on new server
      o   chkconfig ndo2db off

I'd move the disable notifications to the top of the list. I'm pretty certain that bit is stored in the postgresql and will survive backup/restore.

Since you're disabling the networking in vCenter, I'd also consider adding VMware snapshots to my list near the top too. If your db is on a VM I'd personally poweroff prod and db and take a cold VMware snap of that box.

Otherwise, nothing jumps out at me.

[BanditBBS]

Ok, I attempted it but I failed it severely messed up my httpd install so I backed out and brought the old server back online. I then uninstalled XI and httpd from the new box and reinstalled. Everything working fine on the new box now.

Now to reattempt...can I just remark out the apache/httpd section of the backup and restore scripts? as long as I follow the using ssl with XI docs on new server, I haven't made any other changes to httpd. Also, moving from RHEL 5 to 6 may have been part of the issue.

[jdalrymple]

Ok, I attempted it but I failed it severely messed up my httpd install

In what way? Would httpd not start?

can I just remark out the apache/httpd section of the backup and restore scripts

I'd honestly guess you'd be better off taking your configs with you. Did you per-chance grab any bits of the httpd/error_log?

Also, moving from RHEL 5 to 6 may have been part of the issue.

The major Apache changes happened between EL6 and EL7, I wouldn't expect any weirdness on that move.

I would only comment out the httpd bit in the restore_xi.sh - go ahead and take the backups along, that way you have the old files in case you ever did want them.

[BanditBBS]

Ok, did the move again and it is more or less complete. Couple small tasks to finish in the morning and one last issue to investigate.

Here are a couple issues I had to fix after the restore:
1.) Reinstall nagios-plugins from subcomponents
2.) Reinstall nrpe from subcomponents
They were both over-written by the RHEL5 compiled versions and didn't work.

The remaining issue is with check_http. I get this:

Code: Select all

[root@iss-chi-nag05 libexec]# ./check_http tdocs.trinitylogisticsonline.com --ssl -p 8250 -u /content
CRITICAL - Cannot make SSL connection.
139787257436008:error:140773E8:SSL routines:SSL23_GET_SERVER_HELLO:reason(1000):s23_clnt.c:744:

It is only happening with a few(~10) checks, so it can wait, I'm exhausted! If anyone has a clue about that, I'd appreciate a hint!

[tgriep]

Couple of hints could be a firewall blocking the new server or that the plugin is newer and your check needs to be tweaked.
Look at this.

Code: Select all

 -S, --ssl=VERSION
    Connect via SSL. Port defaults to 443. VERSION is optional, and prevents
    auto-negotiation (1 = TLSv1, 2 = SSLv2, 3 = SSLv3).