Multiline log forwarding isn't working right
Posted: Wed Oct 20, 2021 6:38 pm
I'm trying to use the KB article (link below) to forward and process multiline logs. It doesn't seem to be handling the multiline log file data at all, and is catching all other log data (such as from /var/log/messages) instead of the regex. It creates a huge message until the codec can't handle it and creates a tag named multiline_codec_max_lines_reached.
https://support.nagios.com/kb/article/n ... g-727.html
Below is the rsyslog config, the input and some log entries, both simgle and multiple lines. Got any hintas as to why this fails to work?
Thanks
I also tried creating a multiline forwarder using the newer rsyslog format, which also doesn't work, but rsyslog doesn't complain about syntax.
https://support.nagios.com/kb/article/n ... g-727.html
Below is the rsyslog config, the input and some log entries, both simgle and multiple lines. Got any hintas as to why this fails to work?
Thanks
Code: Select all
$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$InputFileName /var/log/mcs/est01/estservice.log
$InputFileTag syslog_multiline:
$InputFileStateFile nls-state-var_log_mcs_estservice.log # Must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template clean,"%rawmsg%"
if $programname == 'syslog_multiline' then @@2001:4888:a00:3154:f0:ff2:0:b01:5544;clean
if $programname == 'syslog_multiline' then stop
Code: Select all
tcp {
codec => multiline {
pattern => '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
negate => true
what => previous
}
port => 5544
type => 'syslog_multiline'
}
Code: Select all
2021-10-20 03:47:06-094 EstServerImpl INFO Listening for connections on port: 4444...
2021-10-20 03:47:06-094 aa9acf14-de85-4fcf-804e-717e2776bb48 SecurityContextImpl DEBUG Created new instance of SecurityContextImpl
2021-10-20 03:47:06-094 aa9acf14-de85-4fcf-804e-717e2776bb48 EstServerSocketConnectionImpl ERROR Exception occurred for this Socket: Error creating inputstream from Socket
java.io.IOException: Error creating inputstream from Socket
at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:203) ~[estservice.jar:?]
at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.runInner(EstServerSocketConnectionImpl.java:135) ~[estservice.jar:?]
at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.run(EstServerSocketConnectionImpl.java:107) [estservice.jar:?]
at com.verizon.mcs.est.server.impl.EstServerImpl$SocketPool.run(EstServerImpl.java:610) [estservice.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: java.io.EOFException: Connection closed by remote host.
at iaik.security.ssl.Utils.a(SourceFile:292) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.ab.e(SourceFile:350) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.y.g(SourceFile:379) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.ap.c(SourceFile:1206) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.ap.a(SourceFile:1941) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.y.d(SourceFile:798) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.SSLTransport.startHandshake(SourceFile:592) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.SSLTransport.getInputStream(SourceFile:679) ~[iaik_ssl.jar:5.2]
at iaik.security.ssl.SSLSocket.getInputStream(SourceFile:417) ~[iaik_ssl.jar:5.2]
at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:200) ~[estservice.jar:?]
... 4 more
2021-10-20 03:47:06-094 EstServerImpl DEBUG Socket closed. Number of connections = 0
2021-10-20 03:47:06-809 EstServerImpl INFO Connection established. Number of connections = 1inetAddress = /2001:4888:a00:3154:f0:9:0:1 - local address = /2001:4888:a00:3154:f0:ff2:0:701
Code: Select all
module(
load="imfile"
mode="inotify"
)
global(
parser.dropTrailingLFOnReception="on"
parser.escapeControlCharactersOnReceive="on"
)
ruleset(name="app_forward") {
action(type="omfwd"
# target="2001:4888:a03:3161:c0:9:0:100"
target="2001:4888:a00:3154:f0:ff2:0:b01"
protocol="tcp"
port="5544"
template="clean"
)
stop
}
#######################################
input(
type="imfile"
ruleset="app_forward"
# tag="estservice.log"
tag="syslog_multiline"
file="/var/log/mcs/est01/estservice.log"
StateFile="nls-state_var_log_mcs_estservice.log"
escapeLF="on"
reopenOnTruncate="on"
deleteStateOnFileDelete="on"
addMetadata="on" # not sure what the metadata contains--/path/and/file?
template(
name="clean"
type="string"
string="%rawmsg%"
)
)