Multiline codec not working

[prepend]

Good morning,

In our old logstash setup I have a multiline filter for processing log4j logfiles. Because Nagios Log Server has multiple workers and the multiline filter isn't thread safe, the filter has to be migrated to an input codec.

I create a new syslog input and made sure my log4j files are send to port 5545 (using the default rsyslog setup)

Code: Select all

syslog {
    type => 'log4j'
    port => 5545
    codec => multiline {
           pattern => "^%{TIMESTAMP_ISO8601}"
           negate => true
           what => "previous"
    }
}

With this input active, the log files don't show up in nagios. When I remove te codec part, they do though

Am I doing something wrong and is somebody experiences the same issue?

Cheers,
Stefan

[scottwilkerson]

This looks almost correct, as long as your beginning items match %{TIMESTAMP_ISO8601}, however you should change from syslog to tcp because the syslog input is going to apply an additional grok pattern.

Something like this should work, I also added a space after the timestamp as there usually is one

Code: Select all

tcp {
    type => 'log4j'
    port => 5545
    codec => multiline {
           pattern => "^%{TIMESTAMP_ISO8601} "
           negate => true
           what => "previous"
    }
}

[prepend]

After changing the input from syslog to tcp the message i'm getting looks like this, including syslog prio, timestamp, host and provided tag:

Code: Select all

<133>Feb 21 10:18:53 vps12345 atlassian-jira 2015-02-21 10:18:52,917 atlassian-scheduler-quartz1.clustered_Worker-3 INFO ServiceRunner     [atlassian.crowd.directory.DbCachingRemoteDirectory] synchronisation for directory [ 10000 ] starting	

On the host I added the logfile to rsyslog using the provided 'setup-linux.sh' file, but now I'm not sure if rsyslog should be used
How whould I send logfiles to NLS over plain tcp, or can I use rsyslog for this too and how whould the configuration be?

Current config:

Code: Select all

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Input for atlassian-jira
$InputFileName /data/jira/logs/atlassian-jira.log
$InputFileTag atlassian-jira
$InputFileStateFile nls-state-data_jira_logs_atlassian-jira.log
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

if $programname == 'atlassian-jira' then @@vps12345:5545
if $programname == 'atlassian-jira' then ~

[tgriep]

I found this on rsyslog's web site.
One @ before the host is UDP, Two @@ before the host is TCP
So it looks like it is already TCP.