Page 1 of 1
nxlog multilining problem
Posted: Thu Aug 25, 2016 6:33 am
by _asp_
Hi,
We use nxlog to shipper logs to logstash. I have an issue with using multiline module.
I have following log:
Code: Select all
23.08.2016 22:00:00: [20740] INFO: Line 1
23.08.2016 22:00:00: [20740] Line 2
23.08.2016 22:00:00: [20740] Line 3
23.08.2016 22:00:00: [20740] Line 4
23.08.2016 22:00:00: [20740] Line 5
23.08.2016 22:00:00: [20745] INFO: Line 1
23.08.2016 22:00:00: [20745] Line 2
23.08.2016 22:00:00: [20745] Line 3
23.08.2016 22:00:00: [20745] Line 4
23.08.2016 22:00:00: [20745] Line 5
Each "multiline log line" is beginning with an empty line. So I tried to use the empty line as header:
Code: Select all
<Extension multilineEmtpyLine>
Module xm_multiline
HeaderLine /^$/
</Extension>
<Input foo>
Module im_file
File "C:/logfile/foo.log"
#enabling multilining
InputType multilineEmtpyLine
SavePos TRUE
Exec $Message = $raw_event;
</Input>
<Output localTCP>
Module om_tcp
Host localhost
Port 5544
Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();
# Uncomment for debug output
Exec file_write('c:\nxlog\nxlog_localtcp_debug_output.log', $raw_event + "\n");
</Output>
<Route nxlogLocal>
#Path topbeat_debug, ttp_debug => localTCP
Path foo=> localTCP
</Route>
As I see in debug output and logstash each source line will be transmitted as single line. Multilining is not working.
How can I get it work?
thanks, Andreas
Re: nxlog multilining problem
Posted: Thu Aug 25, 2016 11:20 am
by mcapra
Is there a particular downside to delimiting on
INFO rather than a newline?
Using the following sample:
Code: Select all
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
23.08.2016 22:00:00: [20740] INFO: ZZZ
23.08.2016 22:00:00: [20740] Syn
23.08.2016 22:00:00: [20740] ZZZ
23.08.2016 22:00:00: [20740] Line 49
23.08.2016 22:00:00: [20740] Line Bar
With the following input configuration:
Code: Select all
<Extension multilineEmtpyLine>
Module xm_multiline
HeaderLine /INFO/
</Extension>
<Input foo>
Module im_file
File '%ROOT%\data\mylog.txt'
#enabling multilining
InputType multilineEmtpyLine
SavePos TRUE
Exec if ($raw_event == '') drop();
Exec $Message = $raw_event;
</Input>
I get the following entries:
2016_08_25_11_20_27_Dashboard_Nagios_Log_Server.png
Re: nxlog multilining problem
Posted: Fri Aug 26, 2016 3:48 am
by _asp_
good idea.
I made it to identify the first line and now the parsing works.
So far so good. But I would like to know, if my understanding / regex is wrong to filter for new line as header, or is it a bug of nxlog?
Re: nxlog multilining problem
Posted: Fri Aug 26, 2016 9:19 am
by mcapra
You're on the right track definitely. The regex is just going to need to say a little bit more than "if new line". It's going to need to say something like "if new line and line doesn't contain anything". I wasn't able to achieve that with the way nxlog handles regex, but i'm also not a regex wizard
