Logstash sending logs to other Logstash

[carlos.mangini]

Folks,

I need to set up a Logstash to receive logs from different servers in a network of a client. But the NLS cluster is on my company's network. Because these networks are distinct, passing throught a VPN, I need to use a Logstash to bridge the networks by sending the logs to the NLS Cluster and respecting their source information such as host and IP.

Could you advise me on the best way to proceed for this scenario?


Thanks.

[carlos.mangini]

One more question...

Let's imagine that 100 servers send the log to the client's dedicated network logstash and this logstash sends the data from the 100 servers to the NLS. How to use this infrastructure, ensuring that when receiving the data in the NLS I will count as a log entry of 100 servers and not 1 only?

[cdienger]

Hi @carlos.mangini,

Are the servers sending syslog or other logs like Windows event logs? I don't believe there's a simple solution for this, but could imagine some relatively straight forward options(syslog output > tcp input > filter to parse) if it's only syslog data.

One question though - is another logstash server needed? If the clients can connect to the vpn couldn't they send directly to the NLS server?

[carlos.mangini]

@cdienger,

This logstash on the dedicated client network needs to send all log types to the NLS cluster (Syslog, Event logs and Network). The solution I found was getting everything in logstash, using the logstash-output-file and sending the data via Filebeat (plugin).

But, I would like to preserve the source data of each equipment, as well as the quantities of systems using the tool (Linux, Windows and Network).

I liked your suggestion regarding sending the logs directly via VPN. Using a logstash as a proxy would be to avoid overloading the system. But apart from that, I see no other advantages in using logstash between client and NLS cluster.

[cdienger]

I was able to come up with something I believe meets your request. Using the output:

Code: Select all

file{
path => '/tmp/test-%{+YYYY-MM-dd}.txt'
}

the input:

Code: Select all

file{
path => '/tmp/test-*.txt'
start_position => 'beginning'
}

and the kv filter:

Code: Select all

kv{
value_split => ":"
field_split => ","
recursive => "false"
exclude_keys => [ "@version", "@timestamp", "{message" ]
allow_duplicate_values => true
trimkey => '"'
trim => '"'
}

Details for each can be found at https://www.elastic.co/guide/en/logstas ... -file.html, https://www.elastic.co/guide/en/logstas ... ugins.html and https://www.elastic.co/guide/en/logstas ... rs-kv.html. Make sure to thoroughly test this out though if you're planning to use this in production.

The output is on box A, I then copied the files from box A, to /tmp on box B. The input and kv filter are on box B, read the file in and modify it slightly to parse it and insert it into the database on box B. Hope this helps !