nxlog multilining problem

[_asp_]

Hi,

We use nxlog to shipper logs to logstash. I have an issue with using multiline module.

I have following log:

Code: Select all

23.08.2016 22:00:00: [20740] INFO: Line 1
23.08.2016 22:00:00: [20740] Line 2
23.08.2016 22:00:00: [20740] Line 3
23.08.2016 22:00:00: [20740] Line 4
23.08.2016 22:00:00: [20740] Line 5

23.08.2016 22:00:00: [20745] INFO: Line 1
23.08.2016 22:00:00: [20745] Line 2
23.08.2016 22:00:00: [20745] Line 3
23.08.2016 22:00:00: [20745] Line 4
23.08.2016 22:00:00: [20745] Line 5

Each "multiline log line" is beginning with an empty line. So I tried to use the empty line as header:

Code: Select all

<Extension multilineEmtpyLine>
    Module xm_multiline
    HeaderLine /^$/
</Extension>

<Input foo>
    Module im_file
    File "C:/logfile/foo.log"
    
    #enabling multilining
    InputType multilineEmtpyLine
    SavePos TRUE
    Exec $Message = $raw_event;
</Input>

<Output localTCP>
    Module om_tcp
    Host localhost
    Port 5544
   
    Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec $raw_event = to_json();
  
    # Uncomment for debug output
    Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n");
</Output>

<Route nxlogLocal>
    #Path topbeat_debug, ttp_debug => localTCP
    Path foo=> localTCP
</Route>

As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working.

How can I get it work?

thanks, Andreas

[mcapra]

Is there a particular downside to delimiting on INFO rather than a newline?

Using the following sample:

Code: Select all

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar

With the following input configuration:

Code: Select all

<Extension multilineEmtpyLine>
    Module xm_multiline
    HeaderLine /INFO/
</Extension>

<Input foo>
    Module im_file
    File '%ROOT%\data\mylog.txt'
    
    #enabling multilining
    InputType multilineEmtpyLine
    SavePos TRUE
	Exec if ($raw_event == '') drop();
	Exec 	$Message = $raw_event;
</Input>

I get the following entries:

2016_08_25_11_20_27_Dashboard_Nagios_Log_Server.png

[_asp_]

good idea.

I made it to identify the first line and now the parsing works.

So far so good. But I would like to know, if my understanding / regex is wrong to filter for new line as header, or is it a bug of nxlog?

[mcapra]

You're on the right track definitely. The regex is just going to need to say a little bit more than "if new line". It's going to need to say something like "if new line and line doesn't contain anything". I wasn't able to achieve that with the way nxlog handles regex, but i'm also not a regex wizard