Defaults missing - again

[Fred Kroeger]

No joy...
There was only one file on node 1 dated 3/5/15 - 4.5MB - it said it restored successfully but no difference

Code: Select all

drwxr-xr-x. 2 nagios nagios    4096 May  3 20:01 .
drwxr-xr-x. 3 nagios nagios    4096 Feb 13 07:24 ..
-rw-r--r--  1 nagios users  4531360 May  3 20:01 nagioslogserver.2015-05-03.1430625557.tar.gz

Node 2 had a couple of zero length files and some very small files - again it restored successfully but no difference
Interestingly these files had been updated on 3rd May even though some had been craeted earlier (going by the datestamp on the filename)

Code: Select all

-rw-r--r--  1 nagios users  651264 May  3 19:45 nagioslogserver.2015-04-29.1430262132.tar.gz
-rw-r--r--  1 nagios users  798720 May  3 19:46 nagioslogserver.2015-04-29.1430262242.tar.gz
-rw-r--r--  1 nagios users       0 May  3 19:45 nagioslogserver.2015-04-30.1430366357.tar.gz
-rw-r--r--  1 nagios users   12288 May  3 19:47 nagioslogserver.2015-05-03.1430653547.tar.gz
-rw-r--r--  1 nagios users       0 May  3 19:47 nagioslogserver.2015-05-03.1430653629.tar.gz

[Fred Kroeger]

Next option -- Can I uninstall NLS and then reinstall it via the latest update ?

Fred

[jolson]

Fred,

That's definitely a possibility. You shouldn't even need to stand up a new box - to uninstall you should shut down important services and remove the relevant directories.

Code: Select all

service elasticsearch stop
service logstash stop
service httpd stop
service crond stop
rm -rf /usr/local/nagioslogserver
rm -rf /var/www/html/nagioslogserver

From here you can install our latest version (R1.4).

[Fred Kroeger]

All looking good now - thanks!
I did have a problem with Node1 again when setting up the alternate datapath and lost the defaults again but no problems were experienced when I changed Node 2.
The only difference between both changes is that on Node 2 I also stopped the logstash service?
I then stoppped both logstash & elasticsearch on Node 1 , copied all the data files again from the original path and all then worked OK. So not sure if stopping the logstash service is also required when moving the elasticsearch data ?

Anyway a short time later the elasticsearch service stopped again on Node1 - messages log showed a Java out of memory message. From that point I could start the elasticsearch service but would stop soon after. I rebooted Node1 and it has been good for 1hr now.

Thanks for your help in getting me back to a stable installation.
You did make a comment earlier about the multilple datapaths having a "software-raid functionality". Can you elaborate on this as my reading of the doco was that data was shared between datapaths and not being duplicated (mirrored) across the datapaths

regards... Fred

[jolson]

So not sure if stopping the logstash service is also required when moving the elasticsearch data ?

This is a good thing to note, though it should not be required since Logstash doesn't touch those data directories, nor does it have anything to do with the information that you've been losing. It may be a coincidence, but it's good to know that this seemed to help resolve your issue.

To elaborate about the 'software-raid' functionality I spoke of, I was referencing this document: https://assets.nagios.com/downloads/nag ... Server.pdf

This configuration will store data in both locations in a sort of software RAID favoring the path with the most available free space.

You can read about this implementation here: https://github.com/elastic/elasticsearch/issues/1356

Essentially, when you define two data-paths your data will be striped across both paths similar to what you'd expect from RAID 0.

messages log showed a Java out of memory message

If you continue to see out of memory messages, elasticsearch might be using up all of the memory in your nodes. Let us know if this behavior continues - we can get it fixed if it does.

Glad to hear you're up and running properly!

[Fred Kroeger]

Thanks - I guess I was getting confused with the acronym RAID - as the first letter stands for "redundant".
There is no redundancy or striping with this setup, the whole file is saved to just one of the drives on the log server.
It seems to work pretty well as both drives consistenly have around the same amount of used space.
You can close this now - regards... Fred