How to move data?

[2evanowen]

Hey guys,

Our /usr directory is becoming very large. We are looking to move only the data from elasticsearch over and keeping all of the configuration files in place.
What is the best way to do this? Which folders will I have to move over? Which configuration files will I need to modify? Which services do I need to stop before doing this?


I was thinking of copying all the data from /usr/local/nagioslogserver/elasticsearch/data to /data1/data, then just making a soft link from /usr/local/nagioslogserver/elasticsearch/data to /data1/data. Would that work? Is that all I have to do?

[jolson]

was thinking of copying all the data from /usr/local/nagioslogserver/elasticsearch/data to /data1/data, then just making a soft link from /usr/local/nagioslogserver/elasticsearch/data to /data1/data. Would that work? Is that all I have to do?

I tested the above in a lab, and it worked like a charm. Full procedure:

Code: Select all

service elasticsearch stop
cd /usr/local/nagioslogserver/elasticsearch
mkdir /data1 ; chown nagios.nagios /data1/
mv data/ /data1/
ln -s /data/data/ data
service elasticsearch start

If elasticsearch is running and notices that it has lost data, it will try rebuilding your database with _nothing_ inside of it. You should be able to perform this procedure in a rolling fashion - one instance at a time until complete.

Let us know if you have any problems, thanks!

[2evanowen]

It looks like it worked, until I log in. Then clicking around I have an error and it looks like most of my logs aren't there:

Screen Shot 2016-01-20 at 2.13.56 PM.png

[2evanowen]

I have only updated one of the nodes so far, should I do the other now?

[jolson]

Give the other one a try as well - and be sure that the 'nagios' user is allowed to write to your new directory properly.

[2evanowen]

Okay I did the other one as well. Still Doesn't seem to be helping...
Both servers have the nagios:nagios for the owner and group....

Last one:

Code: Select all

[owen@barium data1]$ ls -al
total 4
drwxr-xr-x   3 nagios nagios   17 Jan 20 13:55 .
dr-xr-xr-x. 19 root   root   4096 Jan 20 09:55 ..
drwxr-xr-x.  4 nagios nagios   92 Nov 23 15:04 data

New server:

Code: Select all

[owen@calcium data1]$ ls -al
total 4
drwxr-xr-x.  3 nagios nagios   17 Jan 20 15:39 .
dr-xr-xr-x. 19 root   root   4096 Jan 20 09:40 ..
drwxr-xr-x.  4 nagios nagios   92 Nov 23 15:06 data

It looks like calcium is getting logs from barium, but that's the only server:

Screen Shot 2016-01-20 at 4.35.40 PM.png

[2evanowen]

Oh yes,
I am getting this FacetPhaseExecutionException message every once in while on the dashboard:

Screen Shot 2016-01-20 at 4.52.51 PM.png

[jolson]

Is your cluster status green?

Code: Select all

curl 'localhost:9200/_cluster/health?level=indices&pretty'

If not, it's possible that you need to give the cluster some time to distribute data appropriately.

Alternatively, you could try changing the DATA_DIR directive in /etc/sysconfig/elasticsearch:

DATA_DIR="/data1/data"

Give elasticsearch a restart after that and let me know if it helps - this directive will not be overwritten on upgrade.

Jesse

[2evanowen]

I got it working!
My global settings were messed up for some reason and didn't have my cluster hostname in there.

Both ways work for me now, the link and the DATA_DIR.

Thanks for your help!

[bwallace]

Thanks for that update and glad to hear it is working -- we'll lock this thread now and feel free to open another should you require assistance with anything else.