Migrating from ELK to Nagios log server

[stryder77]

Good Morning,

I am interested in NLS and have a few questions. We currently have a ELK log management system using 3 nodes for 3 data centers and 2 of those nodes share a GlusterFS partition. We have had this ELK log management for close to 3 years and have a lot of data I would hate to lose. I am trying to read through and understand what licensing option would best fit my needs as well as what would be a good migration path for moving from my existing system to NLS. We already have NxLog installed on all of our servers.

Our existing ELK cluster was built by a previous employee from scratch. I don't have a lot of knowledge about GlusterFS or ELK but I'm slowly learning. I like what I've read about NLS because of its ease of use and ability to scale easily. I have spent several months trying to make the GlusterFS partition work fixing split brain issues and turning on quorum with no luck fixing the issues with our existing ELK system.

What I need is redundancy and something that I don't spend as much time tweaking the actual server to work as much as able to use it to actually look at what is going on with my other servers and services.

A bit about our set up. We have 3 data centers and in each data center we run a VMWare infrastructure with Compellent SANs. Our VM's run both Linux and Windows Server.

Data center 1 has about 120 Servers
Data center 2 has about 100 Servers
Data center 3 has only 50 servers

Data Center 1 and 2 are on the same network with no vlans but have different physical network isssues between them. So if a router is rebooted at Data Center 1, Data Center 2 will take over essential services but in the case of the GlusterFS partition, it breaks the drive because they become out of sync.
Data Center 3 is in its own network behind a firewall.

My questions are this:

1. What licensing version do I need? 2 instance or 4 instance
2. I read something about a DRBD drive that NLS uses for its clustering? Is this similar to what GlusterFS is? I have had a lot of issues with split brain and syncing issues with the existing GlusterFS partition corrupting log files.
3. What would be our path of migrating off our existing ELK cluster to NLS? (This is very important)
4. Is it possible to save the past 3 years of data stored in our ELK cluster?
5. What support options come with NLS?
6. What minimum system requirements would you recommend for this NLS cluster we would assemble?
7. Does support offer migration help as we move our existing ELK to NLS?

I'm sure several of these questions are answered in the documentation or elsewhere in these forums. I will work on digging through all that but any information support, sales, or other NLS users can provide would be a huge benefit to me.

Thank you very much in advance guys.
Jeff

[mcapra]

1. What licensing version do I need? 2 instance or 4 instance

That's not really a question we can answer since it's contingent on how your infrastructure is laid out, what resources the NLS machines will have available, and the amount of traffic you are expecting to receive.

2. I read something about a DRBD drive that NLS uses for its clustering? Is this similar to what GlusterFS is? I have had a lot of issues with split brain and syncing issues with the existing GlusterFS partition corrupting log files.

The clustering is handled almost exclusively by the elasticsearch engine. NLS doesn't come packed with any DRBD components to my knowledge, but that's certainly a possibility. You're better off asking someone with Linbit about that since they could more specifically address the issues that GlusterFS is currently causing in the context of whether or not DRBD is a solution.

3. What would be our path of migrating off our existing ELK cluster to NLS? (This is very important)

This isn't something we have done officially. I can pretty much guarantee that your Kibana components will be an absolute pain to migrate. The logstash configurations shouldn't be too bad; It might require the installation of some plugins or the altering of filters/inputs/outputs. The elasticsearch migration is the big one. It's definitely feasible to migrate the indices to NLS, but it would take some time and potentially some conversion depending on your requirements for accessing the data.

4. Is it possible to save the past 3 years of data stored in our ELK cluster?

It's pretty impossible to store 3 years worth of *searchable* data for most environments. How elasticsearch works is by having indexes that are *open* and indexes that are *closed*. Open indexes are searchable and cached to RAM, closed indexes are not searchable and are stored on disk. You can always open or close indexes as you please, but there are limitations to how much data you can cache in RAM.

5. What support options come with NLS?

That's a better question for sales, but email/forum/phone support options are available. With email+phone we also offer remote assistance sessions if we cannot efficiently troubleshoot the issue via text.

6. What minimum system requirements would you recommend for this NLS cluster we would assemble?

Again, this is contingent on how your infrastructure is laid out, the amount of traffic you are expecting to receive, and the requirements for retention/search-ability.

7. Does support offer migration help as we move our existing ELK to NLS?

We do not at this time, but the migration itself is certainly something that is within the realm of possibility. It's something you could ask questions about here and get concise answers, but not something we can offer substantial assistance with. This process would require someone who is fairly familiar with elasticsearch for sure. NLS doesn't read from every index; It reads from indices with the name logstash-YYYY-MM-DD. Essentially, you'd need to make sure your indices are split in terms of days rather than the content of the indices and that they match the naming conventions used by NLS.

[stryder77]

Hey mcapra,

Thanks for the reply. I spoke with Paul in sales and he mentioned setting up a Quick Start with NLS to get you in here and look at our existing ELK to see what we have and what is possible. I will work on setting that up and getting back with y'all. I also have a meeting with my boss to discuss his requirements. I do have one other question. How does NLS would with the NXLog and pulling data from linux machines? I have this existing set up and I wonder if I can use that existing infrastructure to instead of having to reconfigure everything.

Appreciate the help

Jeff

[mcapra]

I have this existing [nxlog] set up and I wonder if I can use that existing infrastructure to instead of having to reconfigure everything.

This depends on the output methods supported by your nxlog configuration. We usually recommend om_tcp for the Output module and sending the event as JSON using to_json();. There is some stock configuration provided by NLS that you have access to.

How does NLS would with the NXLog and pulling data from linux machines?

For linux machines, we typically recommend rsyslog and have a setup script that takes care of just about everything on your machine that is sending logs.