NLS or LogStash Proxies

[jaimie.livingston]

I have a need to ship logs from a remote device stack to a NLS cluster over a secure WAN link.
For a variety of reasons, I cannot send the logs directly from the devices to the NLS cluster, and will need to use a log collector/proxy of some type.

What is the recommended NLS compatible solution for log collection and forwarding through a proxy?
Are there any configuration/implementation guides for this?

Thanks,

Jaimie Livingston

[ssax]

There aren't currently any guides or direction for proxies/log collectors at this time.

I found this (old but still good info):

https://github.com/elastic/logstash-forwarder

And this:

https://github.com/elastic/beats/tree/master/filebeat

https://www.elastic.co/beats/filebeat

[jaimie.livingston]

I found that a good rsyslog relay setup works very well. Here's an example config we are using...
Please don't give me any grief about the rsyslog new vs. legacy format...

Code: Select all

##########################################################################################
# RSysLog Configuration
# Version: rsyslogd 8.24.0 minimum
# /etc/rsylog.conf

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# Documentation:   https://www.rsyslog.com 
# Troubleshooting: https://www.rsyslog.com/doc/troubleshoot.html
# Tutorial: https://rainer.gerhards.net/2019/10/rsyslog-beginners-tutorial-series.html
# GitHub:   https://github.com/rsyslog/rsyslog

#### MODULES ####

# MESSAGE SOURCES
# The imjournal module now used as a message source instead of imuxsock.
$ModLoad imuxsock 	# provides support for local system logging (e.g. via logger command)
$ModLoad imjournal 	# provides access to the systemd journal
#$ModLoad imklog 	# reads kernel messages (the same are read from journald)
#$ModLoad immark  	# provides --MARK-- message capability
module(load="imudp") 	# load just once to accept inbound udp from remotes, requires an input
module(load="imtcp") 	# load just once to accept inbound tcp from remotes, requires an input


# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

call relaysyslog		# local system logs are still maintained locally, but also forwarded


#### RSYSLOG RELAY CONFIGURATION ####
#Requires imudp and imtcp modules be loaded
#Combines Rulesets with Inputs (LISTENERS).
# The defined input ports should match the defined logstash input ports on the logserver

## SYSLOG Forwarding Rule ##
# BEGIN FORWARDING RULE #
ruleset (name="relaysyslog") {
	$ActionQueueFileName relay5544r1	#unique name prefix for spool files
	$ActionQueueMaxDiskSpace 1g		#1G limit for spool file size
	$ActionQueueHighWaterMark 8000 		# NAGIOSLOGSERVER
	$ActionQueueLowWaterMark 2000 		# NAGIOSLOGSERVER
	$ActionQueueSaveOnShutdown on		#same queued messages to disk on shutdown
	$ActionQueueType LinkedList		#run asynchronously
	$ActionResumeRetryCount -1		#keep trying to relay to logserver, forever
	#logserver host is: name-or-ip:port
	*.* @@YOURLOGSERVERADDRESS:5544
}
# END OF FORWARDING RULE #
input(type="imudp" port="514" ruleset="relaysyslog")
input(type="imudp" port="5544" ruleset="relaysyslog")
input(type="imtcp" port="514" ruleset="relaysyslog")
input(type="imtcp" port="5544" ruleset="relaysyslog")

## EVENTLOG Forwarding Rule ##
# BEGIN FORWARDING RULE #
ruleset (name="relayeventlog") {
	$ActionQueueFileName relay3515r1	#unique name prefix for spool files
	$ActionQueueHighWaterMark 8000 		# NAGIOSLOGSERVER
	$ActionQueueLowWaterMark 2000 		# NAGIOSLOGSERVER
	$ActionQueueMaxDiskSpace 1g		#1G limit for spool file size
	$ActionQueueSaveOnShutdown on		#same queued messages to disk on shutdown
	$ActionQueueType LinkedList		#run asynchronously
	$ActionResumeRetryCount -1		#keep trying to relay to logserver, forever
	#logserver host is: name-or-ip:port
	*.* @@YOURLOGSERVERADDRESS:3515
}
# END OF FORWARDING RULE #
input(type="imudp" port="3515" ruleset="relayeventlog")
input(type="imtcp" port="3515" ruleset="relayeventlog")

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

[scottwilkerson]

I found that a good rsyslog relay setup works very well. Here's an example config config we are using...
Please don't give me any grief about the rsyslog new vs. legacy format...

Code: Select all

##########################################################################################
# RSysLog Configuration
# Version: rsyslogd 8.24.0 minimum
# /etc/rsylog.conf

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# Documentation:   https://www.rsyslog.com 
# Troubleshooting: https://www.rsyslog.com/doc/troubleshoot.html
# Tutorial: https://rainer.gerhards.net/2019/10/rsyslog-beginners-tutorial-series.html
# GitHub:   https://github.com/rsyslog/rsyslog

#### MODULES ####

# MESSAGE SOURCES
# The imjournal module now used as a message source instead of imuxsock.
$ModLoad imuxsock 	# provides support for local system logging (e.g. via logger command)
$ModLoad imjournal 	# provides access to the systemd journal
#$ModLoad imklog 	# reads kernel messages (the same are read from journald)
#$ModLoad immark  	# provides --MARK-- message capability
module(load="imudp") 	# load just once to accept inbound udp from remotes, requires an input
module(load="imtcp") 	# load just once to accept inbound tcp from remotes, requires an input


# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

call relaysyslog		# local system logs are still maintained locally, but also forwarded


#### RSYSLOG RELAY CONFIGURATION ####
#Requires imudp and imtcp modules be loaded
#Combines Rulesets with Inputs (LISTENERS).
# The defined input ports should match the defined logstash input ports on the logserver

## SYSLOG Forwarding Rule ##
# BEGIN FORWARDING RULE #
ruleset (name="relaysyslog") {
	$ActionQueueFileName relay5544r1	#unique name prefix for spool files
	$ActionQueueMaxDiskSpace 1g		#1G limit for spool file size
	$ActionQueueHighWaterMark 8000 		# NAGIOSLOGSERVER
	$ActionQueueLowWaterMark 2000 		# NAGIOSLOGSERVER
	$ActionQueueSaveOnShutdown on		#same queued messages to disk on shutdown
	$ActionQueueType LinkedList		#run asynchronously
	$ActionResumeRetryCount -1		#keep trying to relay to logserver, forever
	#logserver host is: name-or-ip:port
	*.* @@YOURLOGSERVERADDRESS:5544
}
# END OF FORWARDING RULE #
input(type="imudp" port="514" ruleset="relaysyslog")
input(type="imudp" port="5544" ruleset="relaysyslog")
input(type="imtcp" port="514" ruleset="relaysyslog")
input(type="imtcp" port="5544" ruleset="relaysyslog")

## EVENTLOG Forwarding Rule ##
# BEGIN FORWARDING RULE #
ruleset (name="relayeventlog") {
	$ActionQueueFileName relay3515r1	#unique name prefix for spool files
	$ActionQueueHighWaterMark 8000 		# NAGIOSLOGSERVER
	$ActionQueueLowWaterMark 2000 		# NAGIOSLOGSERVER
	$ActionQueueMaxDiskSpace 1g		#1G limit for spool file size
	$ActionQueueSaveOnShutdown on		#same queued messages to disk on shutdown
	$ActionQueueType LinkedList		#run asynchronously
	$ActionResumeRetryCount -1		#keep trying to relay to logserver, forever
	#logserver host is: name-or-ip:port
	*.* @@YOURLOGSERVERADDRESS:3515
}
# END OF FORWARDING RULE #
input(type="imudp" port="3515" ruleset="relayeventlog")
input(type="imtcp" port="3515" ruleset="relayeventlog")

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

Thanks for sharing @jaimie.livingston!