changing data store location
Posted: Fri Mar 20, 2015 2:41 pm
Hello
I followed the instructions to change the data store path for Nagios Log Server here:
http://assets.nagios.com/downloads/nagi ... Server.pdf
It appears that it has failed though. Here's a summary of the steps I followed:
-stopped elasticsearch each instance of NLS in the cluster
-edited the /etc/sysconfig/elasticsearch file:
-created the correspondent data directory
-copied every unique directory into the new path
-started elasticsearch each instance of NLS in the cluster
Please note that I moved the data stores to a common, NFS-mounted directory. I was running into a space-constraint on our local disk and needed more capacity.
Now elasticsearch will only start on 1 member of the cluster and fails on any other member. From /var/log/elasticearch/5a1a1d31-f1af-43f5-afa6-6679ca68524c.log:
Additionally, on the instance that elasticsearch starts on:
-I see the trial copy warning:
Notice: This trial copy of Nagios Log Server will expire in 60 days. Purchase a License Now or Enter your license key.
Although all instances have a proper license.
-NLS has stopped collecting logs altogether
Please let me know any steps or recommendations I should take to get this back into working order.
Thanks.
I followed the instructions to change the data store path for Nagios Log Server here:
http://assets.nagios.com/downloads/nagi ... Server.pdf
It appears that it has failed though. Here's a summary of the steps I followed:
-stopped elasticsearch each instance of NLS in the cluster
-edited the /etc/sysconfig/elasticsearch file:
Code: Select all
# Elasticsearch data directory
DATA_DIR="/mnt/logstash/nagioslogserver/elasticsearch/data"-copied every unique directory into the new path
Code: Select all
ls -l /mnt/logstash/nagioslogserver/elasticsearch/data/
total 12K
drwxrwxr-x 3 nagios nagios 2048 Dec 2 10:03 5a1a1d31-f1af-43f5-afa6-6679ca68524c
drwxrwxr-x 3 nagios nagios 2048 Jan 20 12:00 afc6a3f6-e502-450b-b569-6f449aac5bde
drwxrwxr-x 3 nagios nagios 2048 Jan 20 12:02 e17ee78b-20ce-409f-b5f2-6c3be9e3d735Please note that I moved the data stores to a common, NFS-mounted directory. I was running into a space-constraint on our local disk and needed more capacity.
Now elasticsearch will only start on 1 member of the cluster and fails on any other member. From /var/log/elasticearch/5a1a1d31-f1af-43f5-afa6-6679ca68524c.log:
Code: Select all
[2015-03-20 15:15:18,957][WARN ][common.jna ] Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out. Increase RLIMIT_MEMLOCK (ulimit).
[2015-03-20 15:15:19,053][INFO ][node ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] version[1.3.2], pid[2977], build[dee175d/2014-08-13T14:29:30Z]
[2015-03-20 15:15:19,053][INFO ][node ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] initializing ...
[2015-03-20 15:15:19,068][INFO ][plugins ] [d215d9a9-fdd7-45d3-bda4-e171adcc7012] loaded [knapsack-1.3.2.0-d5501ef], sites []
[2015-03-20 15:15:19,104][ERROR][bootstrap ] {1.3.2}: Initialization Failed ...
- ElasticsearchIllegalStateException[Failed to obtain node lock, is the following location writable?: [/mnt/logstash/nagioslogserver/elasticsearch/data/5a1a1d31-f1af-43f5-afa6-6679ca68524c]]-I see the trial copy warning:
Notice: This trial copy of Nagios Log Server will expire in 60 days. Purchase a License Now or Enter your license key.
Although all instances have a proper license.
-NLS has stopped collecting logs altogether
Please let me know any steps or recommendations I should take to get this back into working order.
Thanks.