Multiple Data Store Paths

[sweath]

So we've been running NLS for about a month now and as we continue to add additional logging, we are quickly realizing we will need a lot more space. I found this reference in adding a second Data Path (https://assets.nagios.com/downloads/nag ... 1487859922) but after following the instructions, I'm left with a NLS instance that I'm assuming is only gathering local log files. Some of the things that happen are as follows:

No dashboards (all disappear)
User permissions are removed
Can't even get to log-in page (can access pages through existing session)


I'm sure there are a lot of other things that happen, as what appears to me is happening is NLS is creating the new Datastore, but forgetting about all the other information in the other Datastore. I've verified permissions, and have attempted to go through the elasticsearch logs, but as much as I like to pretend I'm good at this type of stuff, I'm not. Does anyone have any advice, thanks!

[mcapra]

Is this a single or multiple instance setup?

Can you share the contents of this file:

Code: Select all

/etc/sysconfig/elasticsearch

[sweath]

It is is a single instance. Attached is my current config. When I add the second Data Store path DATA_DIR looks like such:

DATA_DIR="$ES_HOME/data,/home/Data"

*If you need me too, I can make that change as well and upload that here, but that is the only additional thing I do.

[mcapra]

What are the permissions on /home/Data?

Code: Select all

ls -al /home/Data

I'm left with a NLS instance that I'm assuming is only gathering local log files.

Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?

[sweath]

What are the permissions on /home/Data?

Permissions for /home/Data:

Code: Select all

drwxr-xr-x   3 nagios nagios   49 Apr 21 09:44 Data

I did not see this earlier but the permissions for my default directory are:

Code: Select all

drwxr-xr-x. 3 nagios nagcmd    49 Mar 16 12:24 data

Could this be the issue?

I'm left with a NLS instance that I'm assuming is only gathering local log files.

Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?

This statement was an assumption as if you click on the "Home" tab, there is a line that states "Send More Logs - Receiving logs from x hosts..." When I implement the second path this number changes from 20+ to 1. If I remove the entry "/home/Data" from /etc/sysconfig/elasticsearch, everything goes back to normal.

[mcapra]

Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:

Code: Select all

[root@nls1 ~]# ls -al /usr/local/nagioslogserver/elasticsearch/data
lrwxrwxrwx 1 nagios nagios 11 Nov  1 09:55 /usr/local/nagioslogserver/elasticsearch/data -> /data1/data

Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:

Code: Select all

/var/log/elasticsearch

This command should package them all into the /tmp/43572_1.zip file:

Code: Select all

zip -r /tmp/43572_1.zip /var/log/elasticsearch/*

[sweath]

Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:

I'm not sure I follow you on this as permissions are correct on the /usr/local/... directory, those were performed from NLS's documentation. I replicated the same process for permissions on the /home/Data directory, which mirror the /usr/local/... directory.

Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:

Code: Select all

/var/log/elasticsearch

[Box293]

I've been doing some testing recently and have had the same issue.

How long have you waited after making the change and restarting elasticsearch? We're wondering if there is something happening in the background that hasn't finished yet.

[sweath]

I waited about 15 minutes the other day after making the change. I thought that may have been the issue as well, but if I go to the new directory I can see that the index has already been created.

[mcapra]

It's likely a change in how our default templates for the indices have gradually shifted with newer versions of Elasticsearch. The striping across volumes never worked particularly well in Elasticsearch and was removed in version 2+ for that reason.

I do apologize for the inconvenience. A bug report has been filed for this issue (ID 11654), though i'm not sure if an official fix will ever come since the functionality seems to be long gone in the latest versions of Elasticsearch.