Page 1 of 1
Logstash sending logs to other Logstash
Posted: Thu Jan 04, 2018 5:44 pm
by carlos.mangini
Folks,
I need to set up a Logstash to receive logs from different servers in a network of a client. But the NLS cluster is on my company's network. Because these networks are distinct, passing throught a VPN, I need to use a Logstash to bridge the networks by sending the logs to the NLS Cluster and respecting their source information such as host and IP.
Could you advise me on the best way to proceed for this scenario?
Thanks.
Re: Logstash sending logs to other Logstash
Posted: Fri Jan 05, 2018 12:14 pm
by carlos.mangini
One more question...
Let's imagine that 100 servers send the log to the client's dedicated network logstash and this logstash sends the data from the 100 servers to the NLS. How to use this infrastructure, ensuring that when receiving the data in the NLS I will count as a log entry of 100 servers and not 1 only?
Re: Logstash sending logs to other Logstash
Posted: Fri Jan 05, 2018 1:36 pm
by cdienger
Hi
@carlos.mangini,
Are the servers sending syslog or other logs like Windows event logs? I don't believe there's a simple solution for this, but could imagine some relatively straight forward options(syslog output > tcp input > filter to parse) if it's only syslog data.
One question though - is another logstash server needed? If the clients can connect to the vpn couldn't they send directly to the NLS server?
Re: Logstash sending logs to other Logstash
Posted: Mon Jan 08, 2018 4:05 pm
by carlos.mangini
@cdienger,
This logstash on the dedicated client network needs to send all log types to the NLS cluster (Syslog, Event logs and Network). The solution I found was getting everything in logstash, using the logstash-output-file and sending the data via Filebeat (plugin).
But, I would like to preserve the source data of each equipment, as well as the quantities of systems using the tool (Linux, Windows and Network).
I liked your suggestion regarding sending the logs directly via VPN. Using a logstash as a proxy would be to avoid overloading the system. But apart from that, I see no other advantages in using logstash between client and NLS cluster.
Re: Logstash sending logs to other Logstash
Posted: Tue Jan 09, 2018 11:54 am
by cdienger
I was able to come up with something I believe meets your request. Using the output:
Code: Select all
file{
path => '/tmp/test-%{+YYYY-MM-dd}.txt'
}
the input:
Code: Select all
file{
path => '/tmp/test-*.txt'
start_position => 'beginning'
}
and the kv filter:
Code: Select all
kv{
value_split => ":"
field_split => ","
recursive => "false"
exclude_keys => [ "@version", "@timestamp", "{message" ]
allow_duplicate_values => true
trimkey => '"'
trim => '"'
}
Details for each can be found at
https://www.elastic.co/guide/en/logstas ... -file.html,
https://www.elastic.co/guide/en/logstas ... ugins.html and
https://www.elastic.co/guide/en/logstas ... rs-kv.html. Make sure to thoroughly test this out though if you're planning to use this in production.
The output is on box A, I then copied the files from box A, to /tmp on box B. The input and kv filter are on box B, read the file in and modify it slightly to parse it and insert it into the database on box B. Hope this helps
!