Page 1 of 1
parse log file with grok
Posted: Wed Nov 23, 2016 5:51 am
by DigNetwerk
Hi,
i've got a log file with the following content:
2016-11-23 03:00:14.651 - 00000001;{00000000-0000-0000-0000-000000000000};Print Manager Started
2016-11-23 03:00:14.714 - 00000000;{00000000-0000-0000-0000-000000000000};Synchronization: APS information restored
When i try to parse these lines with grok, i always get a grokparse error, i've come up with the following filter to try to debug:
if [type] == "srvprintrp-momaps" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601} \- %{BASE10NUM}\;\{00000000\-0000\-0000\-0000\-000000000000\}\;%{GREEDYDATA:info1}" ]
add_tag => "grokked_srvprintrp1"
}
grok {
match => [ "message", "%{YEAR:Year}-%{GREEDYDATA:info2}" ]
add_tag => "grokked_srvprintrp2"
}
}
However, the first IF clause always fails,
when i use: match => [ "message", "%{GREEDYDATA:Data}" ]
everything 'works'
when i added the parse the year: match => [ "message", "%{YEAR:Year}-%{GREEDYDATA:Data}" ]
it would fail again.
I have no idea to debug this simple log further?
Re: parse log file with grok
Posted: Wed Nov 23, 2016 10:19 am
by mcapra
It's not entirely clear what you're trying to do with this grok filter. There are some syntax errors and consistency issues throughout.
If you can tell me how you would like to break this message down, I would be happy to assist with writing a grok filter for the use case.
Re: parse log file with grok
Posted: Wed Nov 23, 2016 12:35 pm
by WillemDH
He wants the logs parsed in a few fields. The logs come in as type "srvprintrp-momaps".
Maybe something like this?
Code: Select all
if [type] == "srvprintrp-momaps" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logtimestamp} \- %{BASE10NUM:logsequence}\;\{00000000\-0000\-0000\-0000\-000000000000\:logid}\;%{GREEDYDATA:logmessage}" ]
add_tag => "grokked_srvprintrp1"
}
}
The weird thing is that the logs:
Code: Select all
2016-11-23 03:00:14.651 - 00000001;{00000000-0000-0000-0000-000000000000};Print Manager Started
2016-11-23 03:00:14.714 - 00000000;{00000000-0000-0000-0000-000000000000};Synchronization: APS information restored
match in tools like
http://grokconstructor.appspot.com/do/match
but we keep getting grokfailures.
Grtz
Willem
Re: parse log file with grok
Posted: Wed Nov 23, 2016 12:58 pm
by mcapra
It's probably an issue with escaping.
I had luck with the following filter:
Code: Select all
if [type] == "srvprintrp-momaps" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logtimestamp} - %{BASE10NUM:logsequence};\{(?<logid>[0-9]{8}-[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{12})\};%{GREEDYDATA:logmessage}" ]
add_tag => "grokked_srvprintrp1"
}
}
Which, using the following source message:
Code: Select all
2016-11-23 03:00:14.651 - 00000001;{00000000-0000-0000-0000-000000000000};My test messsage: some other stuff [but also useful]
Produced the following event:
Re: parse log file with grok
Posted: Thu Nov 24, 2016 3:48 am
by DigNetwerk
i'm now using the following grok filter, but still it shows up as 'failed':
Code: Select all
if [type] == "srvprintrp-momaps" {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logtimestamp} - %{BASE10NUM:logsequence};\{(?<logid>[0-9]{8}-[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{12})\};%{GREEDYDATA:logmessage}" ]
add_tag => "grokked_srvprintrp1"
}
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:logtimestamp} - %{GREEDYDATA:Data}" ]
add_tag => "grokked_srvprintrp2"
}
}
I still can't understand why it would also fail on the second grok parse...
Some more info:
The input filter in naglog:
Code: Select all
tcp {
type => 'srvprintrp-momaps'
port => 5612
codec => json {
charset => 'CP1252'
}
}
the nxlog config file: (Note, the logs are in Unicode)
Code: Select all
<Input file2>
Module im_file
File "C:\Program Files (x86)\uniFLOW Remote Print Server\Data\MomAps_*.Log"
ReadFromLast True
SavePos True
Exec $message = $raw_event; to_json();
</Input>
<Output out2>
Module om_tcp
Host 10.54.25.140
Port 5612
#Exec $hostname = hostname(); $raw_event = $Hostname + " " + $raw_event;
Exec $raw_event = to_json();
</Output>
<Route 2>
Path file2 => out2
</Route>
EDIT:
I have changed the nxlog to:
Code: Select all
<Input file2>
Module im_file
Exec convert_fields("UTF-16LE","UTF-8"); if $raw_event == "" drop();
File "C:\Program Files (x86)\uniFLOW Remote Print Server\Data\MomAps_*.Log"
ReadFromLast True
SavePos True
Exec $message = $raw_event; to_json();
</Input>
and it now parses the grok filter, but now i see in the fields SourceModuleName & SourceModuleType, some strange characters.
grokparsefail2.png
Re: parse log file with grok
Posted: Mon Nov 28, 2016 10:16 am
by mcapra
The strange characters are probably a combination of your input rule:
Code: Select all
tcp {
type => 'srvprintrp-momaps'
port => 5612
codec => json {
charset => 'CP1252'
}
}
And your nxlog object definition:
Code: Select all
<Input file2>
Module im_file
Exec convert_fields("UTF-16LE","UTF-8"); if $raw_event == "" drop();
File "C:\Program Files (x86)\uniFLOW Remote Print Server\Data\MomAps_*.Log"
ReadFromLast True
SavePos True
Exec $message = $raw_event; to_json();
</Input>
The input rule for Nagios Log Server is expecting the message to be encoded in CP1252 (
charset => 'CP1252'), but you are converting the fields to UTF-8 in your nxlog definition (
convert_fields("UTF-16LE","UTF-8");). This likely confuses logstash since it is expecting CP1252 but receiving UTF-8.