Tiered NLS Architecture

[animumrege]

We are in the last few weeks of testing NLS before making a purchase decision. However we can't seem to get the architecture we need up and running properly. We need a multi-tiered configuration where devices send data up to a centralized NLS server from multiple disparate locations. I have attached an example of the architecture we need.

We only have 3 input data types.
1.) JSON log file. This lives on a Freebsd 10.3 server
2.) Syslog data. There are Ubuntu and Centos 7 devices where we need to send critical events
3.) Active Directory. We are using Nxlog to capture AD event data and forward to NLS

Our issue right now is that we can get NLS setup in a single tier configuration, however we need each of those locations to report data higher up.

We have tried 2 different scenarios

1.) Using syslog as the relay (red) and sending all data to NLS
2.) Using NLS as the relay and using output filters to send data to the central location.

Neither seems to work properly.

Has anyone had an success with this type of configuration? If not, anyone out there familiar with relay for syslog or json or using NLS?

any help would be greatly appreciated. This is holding up our purchase because the architecture has to be solid before we are allowed to complete the trial.

Thanks

[mcapra]

3 options that come to mind (some of them you've addressed):

A). source(s) -> rsyslog forwarder -> Remote NLS - The most lightweight option that has all of your logs passing through a forwarder (via rsyslog) on their way to the Nagios Log Server cluster.
B). source(s) -> Logstash with Redis -> Remote NLS - Very similar to option B, but adds a Redis intermediary to account for additional redundancy. Redis will provide a buffer for your logs to live in if the remote Nagios Log Server cluster goes down or is otherwise unavailable. Option A would also do this, but Redis is much much better at storing logs for any amount of time than rsyslog is.
C). source(s) -> Local NLS -> Remote NLS - This is useful if you're interested in local collection/analysis of logs.

I've had all of these options working in some form at one point or another. What sorts of problems are you encountering with your proposed setups?

[animumrege]

Really 2 issues.

1.) Data consistency across datatypes. In a single tier, we can get this mostly working. Once we add the top level, transforming json to syslog basically back to json seems to cause issues. The JSON data is an IDS (basically snort) so it natively writes JSON. we would like to transfer JSON from bottom to top and let NLS pick it up automatically so we could just perform filter / event automation at the top layer.

2.) General transmission of data. We arent sure why, but we have gone through 2 quickstart sessions and still can not get the JSON data from the #1 source system (syslog to send JSON file) to get to the NLS node consistently. All our systems are Centos7 or FreeBSD for the syslog to file solution.

Ideally, we like scenario #3 because that would allow local review of logs and a logical separation for people to access data, however when we attempted to add an ouput filter in NLS to send data to another NLS node, no data ever came across. We have disabled all firewalls as well as SElinux for the time being internally on these systems while we troubleshoot.

As a side note, we did do #2 setup (without redis) just setting up ELK (v4 and the newest v5.1.1) ourselves and that seems to work for data transmission/collection. However, we really just dont want to mix / match systems if at all possible and would like to use NLS to perform the solution end to end whether its using syslog relay or NLS as the relay layer of the solution.

Hope this helps.

Thanks

[mcapra]

Ah, I recall doing a Quickstart for Animum.

transforming json to syslog basically back to json seems to cause issues.

I believe the problem with this was that the syslog output plugin was only ever transporting the message field, which was absent from the JSON after it was parsed. I'll play around with this today and get back to you once I have a working setup, but it may take some time. I think the tcp output plugin leveraging the json codec should be the appropriate option.

we have gone through 2 quickstart sessions and still can not get the JSON data from the #1 source system (syslog to send JSON file) to get to the NLS node consistently.

Can you provide me with the filters/inputs being applied to the data as well as some samplings of the data that is causing issues? I'd like to reproduce this.

[animumrege]

Here is a dump of the json file directly from the source. The input filters and output are fortcoming.

[animumrege]

Attach is a text file with filters etc contained within. This is directly from our NLS demo environment.

You will notice the filter relates to modifying the IP etc for geolocation. That appears to be working.

There are 2 outputs. We have tried both of them separately.

[mcapra]

I have a single instance cluster forwarding to a 2-instance cluster with the following output definition:

Code: Select all

elasticsearch {
        cluster => 'a4e1a590-f811-4545-b7a9-c5b160afebc2'
        host => '192.168.67.4'
        document_type => '%{type}'
        protocol => 'transport'
        workers => 4
    }

Where 192.168.67.4 is the "remote AWS NLS cluster" with it's corresponding cluster ID in your case, and the machine this output is defined on would be the local collection/analysis NLS instance. I would suggest altering the elasticsearch output you've configured to include the settings I have and see if that fixes anything.

[animumrege]

Ok. Does NLS natively support communication with ES? I think during our quickstart we attempted this and ES wasnt configured to allow remote systems to connect directly.

[mcapra]

We probably tried this using the http endpoint previously. I'm using the transport protocol instead.

[animumrege]

It appears that the systems are making a connection. However nothing showing up in the dashboards in the destination.

On the relay NLS I ran netstat -na and got the following results.

Source NLS is 192.168.x.112
Destination NLS is 192.168.x.113

Source netstat:
tcp 0 0 192.168.x.112:56378 192.168.x.113:9300 ESTABLISHED
tcp 0 0 192.168.x.112:56368 192.168.x.113:9300 ESTABLISHED
tcp 0 0 192.168.x.112:56406 192.168.x.113:9300 ESTABLISHED
tcp 0 0 192.168.x.112:56462 192.168.x.113:9300 ESTABLISHED
tcp 0 0 192.168.x.112:56370 192.168.4.113:9300 ESTABLISHED

Destination netstat:
tcp6 0 0 192.168.x.113:9300 192.168.x.112:56370 ESTABLISHED
tcp6 0 0 192.168.x.113:9300 192.168.x.112:56462 ESTABLISHED
tcp6 0 0 192.168.x.113:9300 192.168.x.112:56378 ESTABLISHED
tcp6 0 0 192.168.x.113:9300 192.168.x.112:56406 ESTABLISHED
tcp6 0 0 192.168.x.113:9300 192.168.x.112:56368 ESTABLISHED