Multiline log forwarding isn't working right

This support forum board is for support questions relating to Nagios Log Server, our solution for managing and monitoring critical log data.
Locked
gormank
Posts: 1114
Joined: Tue Dec 02, 2014 12:00 pm

Multiline log forwarding isn't working right

Post by gormank »

I'm trying to use the KB article (link below) to forward and process multiline logs. It doesn't seem to be handling the multiline log file data at all, and is catching all other log data (such as from /var/log/messages) instead of the regex. It creates a huge message until the codec can't handle it and creates a tag named multiline_codec_max_lines_reached.
https://support.nagios.com/kb/article/n ... g-727.html

Below is the rsyslog config, the input and some log entries, both simgle and multiple lines. Got any hintas as to why this fails to work?

Thanks

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm

$InputFileName /var/log/mcs/est01/estservice.log
$InputFileTag syslog_multiline:
$InputFileStateFile nls-state-var_log_mcs_estservice.log # Must be unique for each file being polled
$InputFileSeverity info
$InputFilePersistStateInterval 20000
$InputRunFileMonitor
$template clean,"%rawmsg%"

if $programname == 'syslog_multiline' then @@2001:4888:a00:3154:f0:ff2:0:b01:5544;clean
if $programname == 'syslog_multiline' then stop

Code: Select all

tcp {
    codec => multiline {
        pattern => '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
        negate => true
        what => previous
    }
    port => 5544
    type => 'syslog_multiline'
}

Code: Select all

2021-10-20 03:47:06-094    EstServerImpl INFO Listening for connections on port: 4444...
2021-10-20 03:47:06-094 aa9acf14-de85-4fcf-804e-717e2776bb48   SecurityContextImpl DEBUG Created new instance of SecurityContextImpl
2021-10-20 03:47:06-094 aa9acf14-de85-4fcf-804e-717e2776bb48   EstServerSocketConnectionImpl ERROR Exception occurred for this Socket: Error creating inputstream from Socket
java.io.IOException: Error creating inputstream from Socket
        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:203) ~[estservice.jar:?]
        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.runInner(EstServerSocketConnectionImpl.java:135) ~[estservice.jar:?]
        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.run(EstServerSocketConnectionImpl.java:107) [estservice.jar:?]
        at com.verizon.mcs.est.server.impl.EstServerImpl$SocketPool.run(EstServerImpl.java:610) [estservice.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
Caused by: java.io.EOFException: Connection closed by remote host.
        at iaik.security.ssl.Utils.a(SourceFile:292) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.ab.e(SourceFile:350) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.y.g(SourceFile:379) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.ap.c(SourceFile:1206) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.ap.a(SourceFile:1941) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.y.d(SourceFile:798) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.SSLTransport.startHandshake(SourceFile:592) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.SSLTransport.getInputStream(SourceFile:679) ~[iaik_ssl.jar:5.2]
        at iaik.security.ssl.SSLSocket.getInputStream(SourceFile:417) ~[iaik_ssl.jar:5.2]
        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:200) ~[estservice.jar:?]
        ... 4 more
2021-10-20 03:47:06-094    EstServerImpl DEBUG Socket closed. Number of connections = 0
2021-10-20 03:47:06-809    EstServerImpl INFO Connection established. Number of connections = 1inetAddress = /2001:4888:a00:3154:f0:9:0:1 - local address = /2001:4888:a00:3154:f0:ff2:0:701
I also tried creating a multiline forwarder using the newer rsyslog format, which also doesn't work, but rsyslog doesn't complain about syntax.

Code: Select all

module(
  load="imfile"
  mode="inotify"
)

global(
  parser.dropTrailingLFOnReception="on"
  parser.escapeControlCharactersOnReceive="on"
)

ruleset(name="app_forward") {
  action(type="omfwd"
#    target="2001:4888:a03:3161:c0:9:0:100"
    target="2001:4888:a00:3154:f0:ff2:0:b01"
    protocol="tcp"
    port="5544"
    template="clean"
  )
  stop
}
#######################################

input(
  type="imfile"
  ruleset="app_forward"
#  tag="estservice.log"
  tag="syslog_multiline"
  file="/var/log/mcs/est01/estservice.log"
  StateFile="nls-state_var_log_mcs_estservice.log"
  escapeLF="on"
  reopenOnTruncate="on"
  deleteStateOnFileDelete="on"
  addMetadata="on"    # not sure what the metadata contains--/path/and/file?

  template(
    name="clean" 
	type="string" 
	string="%rawmsg%"
  )
)
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Multiline log forwarding isn't working right

Post by ssax »

Try changing it to this (adding max_lines to be higher than the number of lines that will be sent):

Code: Select all

tcp {
    codec => multiline {
        pattern => '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
        negate => true
        what => previous
        max_lines => 1000
    }
    port => 5544
    type => 'syslog_multiline'
}
gormank
Posts: 1114
Joined: Tue Dec 02, 2014 12:00 pm

Re: Multiline log forwarding isn't working right

Post by gormank »

The issue here isn't the number of lines since the multiline log data is pretty small, and it isn't from the host with the multiline rsyslog forwarder.
What's happening is that data from the host with the multiline config is not being processed (or not received) at all. Hosts where there is no multiline config (basically the clean template) is being globbed into one message by the multiline input on NLS.
So odd as it seems, the input is dropping (or not receiving) all data from the host with the multiline config, and processing (putting many lines into one message) all the data from hosts that don't have any multiline config.
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Multiline log forwarding isn't working right

Post by ssax »

Now I understand, I saw that error assumed too much.

I'm going to lab this up and will update soon.
gormank
Posts: 1114
Joined: Tue Dec 02, 2014 12:00 pm

Re: Multiline log forwarding isn't working right

Post by gormank »

Any luck on this one?
Thanks!
ssax
Dreams In Code
Posts: 7682
Joined: Wed Feb 11, 2015 12:54 pm

Re: Multiline log forwarding isn't working right

Post by ssax »

Removing the ^ makes it work for me:

Code: Select all

tcp {
    codec => multiline {
        pattern => '[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
        negate => true
        what => previous
        max_lines => 1000
    }
    port => 5544
    type => 'syslog_multiline'
}
But the message in Log Server shows up like this:

Code: Select all

<133>Nov 10 18:21:13 localhost syslog_multiline: 2021-11-10 03:47:06-094 aa9acf14-de85-4fcf-804e-717e2776bb48   EstServerSocketConnectionImpl ERROR Exception occurred for this Socket: Error creating inputstream from Socket
<133>Nov 10 18:21:13 localhost syslog_multiline: java.io.IOException: Error creating inputstream from Socket
<133>Nov 10 18:21:13 localhost syslog_multiline:        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:203) ~[estservice.jar:?]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.runInner(EstServerSocketConnectionImpl.java:135) ~[es
tservice.jar:?]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.run(EstServerSocketConnectionImpl.java:107) [estservice.jar:?]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at com.verizon.mcs.est.server.impl.EstServerImpl.run(EstServerImpl.java:610) [estservice.jar:?]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_302]
<133>Nov 10 18:21:13 localhost syslog_multiline: Caused by: java.io.EOFException: Connection closed by remote host.
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.Utils.a(SourceFile:292) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.ab.e(SourceFile:350) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.y.g(SourceFile:379) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.ap.c(SourceF
ile:1206) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.ap.a(SourceFile:1941) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.y.d(SourceFile:798) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.SSLTransport.startHandshake(SourceFile:592) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.SSLTransport.getInputStream(SourceFile:679) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at iaik.security.ssl.SSLSocket.getInputStream(SourceFile:417) ~[iaik_ssl.jar:5.2]
<133>Nov 10 18:21:13 localhost syslog_multiline:        at com.verizon.mcs.est.server.impl.EstServerSocketConnectionImpl.setupInputStream(EstServerSocketConnectionImpl.java:200) ~[estservice.jar:?]
<133>Nov 10 18:21:13 localhost syslog_multiline:        ... 4 more
Rsyslog looks like it is adding <133>Nov 10 18:21:13 localhost syslog_multiline: onto the beginning but I haven't yet determined how to make that not show up.
Locked