changing data store location

[zakhemj]

Hello

I followed the instructions to change the data store path for Nagios Log Server here:

http://assets.nagios.com/downloads/nagi ... Server.pdf

It appears that it has failed though. Here's a summary of the steps I followed:

-stopped elasticsearch each instance of NLS in the cluster
-edited the /etc/sysconfig/elasticsearch file:

Code: Select all

# Elasticsearch data directory
DATA_DIR="/mnt/logstash/nagioslogserver/elasticsearch/data"

-created the correspondent data directory
-copied every unique directory into the new path

Code: Select all

ls -l /mnt/logstash/nagioslogserver/elasticsearch/data/
total 12K
drwxrwxr-x 3 nagios nagios 2048 Dec  2 10:03 5a1a1d31-f1af-43f5-afa6-6679ca68524c
drwxrwxr-x 3 nagios nagios 2048 Jan 20 12:00 afc6a3f6-e502-450b-b569-6f449aac5bde
drwxrwxr-x 3 nagios nagios 2048 Jan 20 12:02 e17ee78b-20ce-409f-b5f2-6c3be9e3d735

-started elasticsearch each instance of NLS in the cluster

Please note that I moved the data stores to a common, NFS-mounted directory. I was running into a space-constraint on our local disk and needed more capacity.

Now elasticsearch will only start on 1 member of the cluster and fails on any other member. From /var/log/elasticearch/5a1a1d31-f1af-43f5-afa6-6679ca68524c.log:

Code: Select all

[2015-03-20 15:15:18,957][WARN ][common.jna               ] Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).
[2015-03-20 15:15:19,053][INFO ][node                     ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] version[1.3.2], pid[2977], build[dee175d/2014-08-13T14:29:30Z]
[2015-03-20 15:15:19,053][INFO ][node                     ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] initializing ...
[2015-03-20 15:15:19,068][INFO ][plugins                  ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] loaded [knapsack-1.3.2.0-d5501ef], sites []
[2015-03-20 15:15:19,104][ERROR][bootstrap                ] {1.3.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/mnt/logstash/nagioslogserver/elasticsearch/data/5a1a1d31-f1af-43f5-afa6-6679ca68524c]]

Additionally, on the instance that elasticsearch starts on:

-I see the trial copy warning:
Notice: This trial copy of Nagios Log Server will expire in 60 days. Purchase a License Now or Enter your license key.

Although all instances have a proper license.

-NLS has stopped collecting logs altogether

Please let me know any steps or recommendations I should take to get this back into working order.
Thanks.

[scottwilkerson]

You cannot have multiple instances write to the same directory.