Database problems after moving backup to new cluster

[Jklre]

We are getting ready to upgrade Nagios Log server to the latest version and migrate it to a new cluster before decommissioning the old one. I went ahead and installed the latest version and restored a backup of the old database. I then activated the licence key. I am running into a few issues.

1) the database appears to be missing any and all indices

2) Logstash does not appear to be collecting any logs.

Code: Select all

9,740,000
Documents
19
Total Shards
18
Successful Shards
0
Indices
1.1GB
Primary Size
1.2GB
Total Size

Code: Select all

Status	Red
Timed Out?	false
# Instances	2
# Data Instances	2
Active Primary Shards	11
Active Shards	18
Relocating Shards	0
Initializing Shards	1
Unassigned Shards	0

For logstash I try to send some test logs and I get connection refused.

do echo "$i" | nc -u -v unls03lxv.corp.int 514 ; done < sendstuffover
nc: Write error: Connection refused

Thank you.

[cdienger]

What steps did you take to restore the database? Are the new nodes members of the old cluster?

To move to a new cluster I generally recommend just adding new nodes one at a time, allowing them to sync, and remove the old nodes - again one at a time and allowing the cluster time to resync after a node's been removed.

One important note is that closed indices will not replicate. You'll need to open closed indices to allow them to replicate. A script to open closed indices to allow them to replicate can be found at https://github.com/elastic/elasticsearch/issues/12963. An example of steps would be:

-add a node
-run the script
-wait for the status to go green
-add second node
-run the script
-wait for the status to go green
-remove old node
-wait for status to go green
-remove second old node
-wait for status to go green

After this the data should be migrated.

[Jklre]

Thanks for the response.

I restored the database using the following steps:

sudo /usr/local/nagioslogserver/scripts/restore_backup.sh /store/backups/nagioslogserver/nagioslogserver.2019-04-10.1554937201.tar.gz

A few things about adding the new nodes to the cluster and syncing it that way. They are different versions. the old cluster is Nagios Log Server 1.4.4 and the new cluster is Nagios Log Server 2.0.7

Is there a way to wipe or do a reset on the database on the new nodes before attempting to sync with the existing cluster? I wouldn't want to corrupt any of the nodes that are in production. Ideally we would like to run 2 clusters in tandem before taking down the existing node to verify its stable.

[cdienger]

restore_backup.sh will restore the part of the database responsible for holding settings - users, dashboards, etc... The actual log data is stored elsewhere.

Do the 1.4.4 machines have a repo configured(Administration > System > Backup & Maintenance) ? If so, we may be able to ditch the idea of clustering them all together and continue to run two clusters separately, and when you're ready to cut over you can just configure the new cluster to use the same repo and pull in data from the there. There's a bit more to it than that but we can get into that later - we should first determine why the new cluster isn't accepting data. Did the restore script import the logstash config? Port 514 is considered a privileged port and there are some extra steps needed to open it up. See: https://assets.nagios.com/downloads/nag ... Server.pdf.

Please PM me a profile if this doesn't resolve the problem of not receiving data. The profile can be generated under Admin > System > System Status.