Updating from CentOS 6 to CentOS 7

[rferebee]

Hello,

We are pushing to get all of our Linux servers on CentOS 7 before the end of the year. Two of my three Log Server boxes are CentOS 6, so I need to upgrade them.

I have an idea of how I'd like to accomplish this task, but I wonder if you could provide your opinion.

1. Configure a new CentOS 7 server with a temporary IP address.
2. Install the latest version of Log Server onto the newly created CentOS 7 server.
3. Record the network information for the CentOS 6 server we are replacing.
4. Shutdown the CentOS 6 server and assign it's network configuration to the CentOS 7 server.
5. Reboot the Log Server environment and allow the index data to copy over to the newly added CentOS 7 server.

I've done the above with a brand new node, but I've never replaced an existing node. Does this sound like it will work or should I consider a different approach? We're looking for as little down time as possible.

My biggest question right now is I have the CentOS 7 server built and I installed Log Server, but when I hit the web interface it's asking me if I want to Install or Connect. Do I even need to worry about that? I'm thinking when I re-IP the server it'll just consider itself already a part of the cluster and I won't have to do anything else. I'm not sure though.

Thank you!

[cdienger]

You'll want to add the nodes to the existing cluster one at a time. The steps would be:

1. install NLS on a machine with a temporary ip
2. add it to the cluster(select 'Connect' in the web UI)
3. allow the status to go green
4. remove one of the old nodes(starting on page 3 of https://support.nagios.com/kb/article/c ... s-549.html)
5. change the IP of the new machine to the one of the old machine - https://support.nagios.com/kb/article/c ... s-549.html
6. repeat for the second new node and remove the remaining old node

One important note is that closed indices will not replicate. You'll need to open closed indices to allow them to replicate. A script to open closed indices to allow them to replicate can be found at https://github.com/elastic/elasticsearch/issues/12963. If there are closed indices then I would add an additional step after step 3 - run the script from the github link and wait for the status to go green again.

[rferebee]

Ok, we have the new node updated with the old node's network information.

The Instance State screen is still showing the old node as being part of the cluster though. There is a little trash can icon I can click to remove it, should I do that?

Also, the hostname is showing up incorrectly for the new node even though we changed it. Will that update on it's own or is there something manual I have to do?

Thank you.

[scottwilkerson]

The Instance State screen is still showing the old node as being part of the cluster though. There is a little trash can icon I can click to remove it, should I do that?

This is the exact reason the icon is there.

If you are done using that instance you can click that icon.

[rferebee]

Ok, I just wanted to make sure before I did it. I wasn't in the instruction provided by the other tech.

Also, just FYI for future updates, there's a typo in the message when you hover over the trash can. It says 'Remove instance form database' instead of 'Remove instance from database'.

[scottwilkerson]

Ok, I just wanted to make sure before I did it. I wasn't in the instruction provided by the other tech.

Also, just FYI for future updates, there's a typo in the message when you hover over the trash can. It says 'Remove instance form database' instead of 'Remove instance from database'.

thanks for the heads up on the typo, we will have that fixed in the next release

[rferebee]

So, everything was looking good yesterday, but I came in this morning and the system took a 1TB snapshot, filled up my storage repository and all the snapshots are missing from the GUI.

All the curator files are still present, but it's not showing the snapshots on the Snapshots & Maintenance page.

Any way I can get them to show back up?

[scottwilkerson]

Can you show the output of this (where /mnt/logstore is the actual path to your snapshot dir)

Code: Select all

ls -ld /mnt/logstore
ll /mnt/logstore

[rferebee]

Here you go. Thanks for your help.

[rferebee@nagioslscc2 ~]$ sudo su -
[root@nagioslscc2 ~]# ls -ld /nlsrepcc
drwxrwx--- 1 nagios nagios 32768 Aug 15 22:32 /nlsrepcc
[root@nagioslscc2 ~]# ll /nlsrepcc
total 19
-rwxrwx--- 0 nagios nagios 192 Aug 16 02:14 index
drwxrwx--- 0 nagios nagios 0 Aug 15 22:32 indices
-rwxrwx--- 0 nagios nagios 504 Jan 29 2019 metadata-curator-20190129123136
-rwxrwx--- 0 nagios nagios 504 Feb 2 2019 metadata-curator-20190202123046
-rwxrwx--- 0 nagios nagios 524 Feb 19 04:30 metadata-curator-20190219123043
-rwxrwx--- 0 nagios nagios 504 Mar 1 22:30 metadata-curator-20190302063044
-rwxrwx--- 0 nagios nagios 522 Apr 2 09:14 metadata-curator-20190402161441
-rwxrwx--- 0 nagios nagios 522 Apr 21 22:30 metadata-curator-20190422053050
-rwxrwx--- 0 nagios nagios 522 May 14 22:31 metadata-curator-20190515053118
-rwxrwx--- 0 nagios nagios 522 Jun 2 22:31 metadata-curator-20190603053119
-rwxrwx--- 0 nagios nagios 525 Jun 23 22:31 metadata-curator-20190624053125
-rwxrwx--- 0 nagios nagios 522 Jul 14 22:31 metadata-curator-20190715053146
-rwxrwx--- 0 nagios nagios 522 Aug 4 22:32 metadata-curator-20190805053209
-rwxrwx--- 0 nagios nagios 522 Aug 14 22:32 metadata-curator-20190815053239
-rwxrwx--- 0 nagios nagios 522 Aug 15 22:32 metadata-curator-20190816053244
-rwxrwx--- 0 nagios nagios 371 Jan 30 2019 snapshot-curator-20190129123136
-rwxrwx--- 0 nagios nagios 361 Feb 2 2019 snapshot-curator-20190202123046
-rwxrwx--- 0 nagios nagios 364 Feb 19 05:41 snapshot-curator-20190219123043
-rwxrwx--- 0 nagios nagios 330 Mar 2 03:28 snapshot-curator-20190302063044
-rwxrwx--- 0 nagios nagios 483 Apr 2 12:47 snapshot-curator-20190402161441
-rwxrwx--- 0 nagios nagios 316 Apr 21 23:01 snapshot-curator-20190422053050
-rwxrwx--- 0 nagios nagios 320 May 14 23:52 snapshot-curator-20190515053118
-rwxrwx--- 0 nagios nagios 316 Jun 2 23:04 snapshot-curator-20190603053119
-rwxrwx--- 0 nagios nagios 313 Jun 23 23:07 snapshot-curator-20190624053125
-rwxrwx--- 0 nagios nagios 523 Jul 14 23:03 snapshot-curator-20190715053146
-rwxrwx--- 0 nagios nagios 315 Aug 4 23:08 snapshot-curator-20190805053209
-rwxrwx--- 0 nagios nagios 319 Aug 14 23:21 snapshot-curator-20190815053239
-rwxrwx--- 0 nagios nagios 0 Aug 16 02:14 snapshot-curator-20190816053244

[scottwilkerson]

There is a corrupt snapshot that will prevent the backups from appearing. You can delete this which is 0 bytes in size:

Code: Select all

rm /nlsrepcc/snapshot-curator-20190816053244