Page 1 of 2
Multiple Data Store Paths
Posted: Fri Apr 21, 2017 10:46 am
by sweath
So we've been running NLS for about a month now and as we continue to add additional logging, we are quickly realizing we will need a lot more space. I found this reference in adding a second Data Path (
https://assets.nagios.com/downloads/nag ... 1487859922) but after following the instructions, I'm left with a NLS instance that I'm assuming is only gathering local log files. Some of the things that happen are as follows:
No dashboards (all disappear)
User permissions are removed
Can't even get to log-in page (can access pages through existing session)
I'm sure there are a lot of other things that happen, as what appears to me is happening is NLS is creating the new Datastore, but forgetting about all the other information in the other Datastore. I've verified permissions, and have attempted to go through the elasticsearch logs, but as much as I like to pretend I'm good at this type of stuff, I'm not. Does anyone have any advice, thanks!
Re: Multiple Data Store Paths
Posted: Fri Apr 21, 2017 1:03 pm
by mcapra
Is this a single or multiple instance setup?
Can you share the contents of this file:
Re: Multiple Data Store Paths
Posted: Fri Apr 21, 2017 1:44 pm
by sweath
It is is a single instance. Attached is my current config. When I add the second Data Store path DATA_DIR looks like such:
DATA_DIR="$ES_HOME/data,/home/Data"
*If you need me too, I can make that change as well and upload that here, but that is the only additional thing I do.
Re: Multiple Data Store Paths
Posted: Fri Apr 21, 2017 2:21 pm
by mcapra
What are the permissions on /home/Data?
I'm left with a NLS instance that I'm assuming is only gathering local log files.
Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?
Re: Multiple Data Store Paths
Posted: Fri Apr 21, 2017 2:34 pm
by sweath
mcapra wrote:What are the permissions on /home/Data?
Permissions for /home/Data:
Code: Select all
drwxr-xr-x 3 nagios nagios 49 Apr 21 09:44 Data
I did not see this earlier but the permissions for my default directory are:
Code: Select all
drwxr-xr-x. 3 nagios nagcmd 49 Mar 16 12:24 data
Could this be the issue?
I'm left with a NLS instance that I'm assuming is only gathering local log files.
Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?
This statement was an assumption as if you click on the "Home" tab, there is a line that states "Send More Logs - Receiving logs from x hosts..." When I implement the second path this number changes from 20+ to 1. If I remove the entry "/home/Data" from /etc/sysconfig/elasticsearch, everything goes back to normal.
Re: Multiple Data Store Paths
Posted: Mon Apr 24, 2017 10:03 am
by mcapra
Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:
Code: Select all
[root@nls1 ~]# ls -al /usr/local/nagioslogserver/elasticsearch/data
lrwxrwxrwx 1 nagios nagios 11 Nov 1 09:55 /usr/local/nagioslogserver/elasticsearch/data -> /data1/data
Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:
This command should package them all into the
/tmp/43572_1.zip file:
Code: Select all
zip -r /tmp/43572_1.zip /var/log/elasticsearch/*
Re: Multiple Data Store Paths
Posted: Tue Apr 25, 2017 8:44 am
by sweath
mcapra wrote:Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:
I'm not sure I follow you on this as permissions are correct on the /usr/local/... directory, those were performed from NLS's documentation. I replicated the same process for permissions on the /home/Data directory, which mirror the /usr/local/... directory.
Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:
Re: Multiple Data Store Paths
Posted: Tue Apr 25, 2017 4:33 pm
by Box293
I've been doing some testing recently and have had the same issue.
How long have you waited after making the change and restarting elasticsearch? We're wondering if there is something happening in the background that hasn't finished yet.
Re: Multiple Data Store Paths
Posted: Tue Apr 25, 2017 5:06 pm
by sweath
I waited about 15 minutes the other day after making the change. I thought that may have been the issue as well, but if I go to the new directory I can see that the index has already been created.
Re: Multiple Data Store Paths
Posted: Wed Apr 26, 2017 10:23 am
by mcapra
It's likely a change in how our default templates for the indices have gradually shifted with newer versions of Elasticsearch. The striping across volumes never worked particularly well in Elasticsearch and was removed in version 2+ for that reason.
I do apologize for the inconvenience. A bug report has been filed for this issue (ID 11654), though i'm not sure if an official fix will ever come since the functionality seems to be long gone in the latest versions of Elasticsearch.