Multiple Data Store Paths
Multiple Data Store Paths
So we've been running NLS for about a month now and as we continue to add additional logging, we are quickly realizing we will need a lot more space. I found this reference in adding a second Data Path (https://assets.nagios.com/downloads/nag ... 1487859922) but after following the instructions, I'm left with a NLS instance that I'm assuming is only gathering local log files. Some of the things that happen are as follows:
No dashboards (all disappear)
User permissions are removed
Can't even get to log-in page (can access pages through existing session)
I'm sure there are a lot of other things that happen, as what appears to me is happening is NLS is creating the new Datastore, but forgetting about all the other information in the other Datastore. I've verified permissions, and have attempted to go through the elasticsearch logs, but as much as I like to pretend I'm good at this type of stuff, I'm not. Does anyone have any advice, thanks!
No dashboards (all disappear)
User permissions are removed
Can't even get to log-in page (can access pages through existing session)
I'm sure there are a lot of other things that happen, as what appears to me is happening is NLS is creating the new Datastore, but forgetting about all the other information in the other Datastore. I've verified permissions, and have attempted to go through the elasticsearch logs, but as much as I like to pretend I'm good at this type of stuff, I'm not. Does anyone have any advice, thanks!
Re: Multiple Data Store Paths
Is this a single or multiple instance setup?
Can you share the contents of this file:
Can you share the contents of this file:
Code: Select all
/etc/sysconfig/elasticsearch
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Multiple Data Store Paths
It is is a single instance. Attached is my current config. When I add the second Data Store path DATA_DIR looks like such:
DATA_DIR="$ES_HOME/data,/home/Data"
*If you need me too, I can make that change as well and upload that here, but that is the only additional thing I do.
DATA_DIR="$ES_HOME/data,/home/Data"
*If you need me too, I can make that change as well and upload that here, but that is the only additional thing I do.
You do not have the required permissions to view the files attached to this post.
Re: Multiple Data Store Paths
What are the permissions on /home/Data?
Code: Select all
ls -al /home/Data
Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?I'm left with a NLS instance that I'm assuming is only gathering local log files.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Multiple Data Store Paths
Permissions for /home/Data:mcapra wrote:What are the permissions on /home/Data?I did not see this earlier but the permissions for my default directory are:Code: Select all
drwxr-xr-x 3 nagios nagios 49 Apr 21 09:44 Data
Could this be the issue?Code: Select all
drwxr-xr-x. 3 nagios nagcmd 49 Mar 16 12:24 data
I'm left with a NLS instance that I'm assuming is only gathering local log files.Could you also expand on this? Unless the old directory was wiped clean, I can't imagine everything would just disappear. If you remove the second path and just use the default path, do all the dashboards and everything else come back?
This statement was an assumption as if you click on the "Home" tab, there is a line that states "Send More Logs - Receiving logs from x hosts..." When I implement the second path this number changes from 20+ to 1. If I remove the entry "/home/Data" from /etc/sysconfig/elasticsearch, everything goes back to normal.
Re: Multiple Data Store Paths
Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:
Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:
This command should package them all into the /tmp/43572_1.zip file:
Code: Select all
[root@nls1 ~]# ls -al /usr/local/nagioslogserver/elasticsearch/data
lrwxrwxrwx 1 nagios nagios 11 Nov 1 09:55 /usr/local/nagioslogserver/elasticsearch/data -> /data1/data
Code: Select all
/var/log/elasticsearch
Code: Select all
zip -r /tmp/43572_1.zip /var/log/elasticsearch/*
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Multiple Data Store Paths
I'm not sure I follow you on this as permissions are correct on the /usr/local/... directory, those were performed from NLS's documentation. I replicated the same process for permissions on the /home/Data directory, which mirror the /usr/local/... directory.mcapra wrote:Can you try giving the nagios.nagios user.group write permissions on that path? Here's my symlinked data path as an example:
Otherwise, I suspect the Elasticsearch logs will show some shard assignment failures. Can you share them? They're typically found here:Code: Select all
/var/log/elasticsearch
You do not have the required permissions to view the files attached to this post.
- Box293
- Too Basu
- Posts: 5126
- Joined: Sun Feb 07, 2010 10:55 pm
- Location: Deniliquin, Australia
- Contact:
Re: Multiple Data Store Paths
I've been doing some testing recently and have had the same issue.
How long have you waited after making the change and restarting elasticsearch? We're wondering if there is something happening in the background that hasn't finished yet.
How long have you waited after making the change and restarting elasticsearch? We're wondering if there is something happening in the background that hasn't finished yet.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Multiple Data Store Paths
I waited about 15 minutes the other day after making the change. I thought that may have been the issue as well, but if I go to the new directory I can see that the index has already been created.
Re: Multiple Data Store Paths
It's likely a change in how our default templates for the indices have gradually shifted with newer versions of Elasticsearch. The striping across volumes never worked particularly well in Elasticsearch and was removed in version 2+ for that reason.
I do apologize for the inconvenience. A bug report has been filed for this issue (ID 11654), though i'm not sure if an official fix will ever come since the functionality seems to be long gone in the latest versions of Elasticsearch.
I do apologize for the inconvenience. A bug report has been filed for this issue (ID 11654), though i'm not sure if an official fix will ever come since the functionality seems to be long gone in the latest versions of Elasticsearch.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/