Page 1 of 1
Help with filtering a custom log
Posted: Tue Jan 13, 2015 8:10 pm
by FelixForbes
Hi There,
I have managed to get several logs from Squid proxies and IIS servers coming into my NLS setup nicely and am starting to look at including custom logs. I have one log which produces output similar to the following:
Code: Select all
2015-01-09 12:36:00,051 [13] INFO [DOMAIN\userid] Closing put queue
I have tried several ways to recognise the fields in the log file. One option I tried was this in my nxlog.conf:
Code: Select all
<Extension w3c>
Module xm_csv
Fields $date, $time, $number, $loglevel, $user, $logmessage
FieldTypes string, string, string, string, string
Delimiter ' '
</Extension>
<Input MQLog>
Module im_file
File "D:\\Logs\\MQApplicationLog.log"
SavePos TRUE
Exec if $raw_event =~ /^#/ drop(); \
else \
{ \
w3c->parse_csv(); \
$SourceName = "MQLog"; \
$Message = to_json(); \
}
</Input>
When I look on the console I only see the default fields with all of the data in the message field. Is there something easy that I am missing in the nxlog.conf? I realise the regex is very basic - I am just trying to get something working before I change the field types and remove excess characters etc.
I have also looked at adding a filter via the Global configuration but I wasn't able to get that working either. I have looked through countless nxlog.conf examples but just can't see what I am missing.
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 4:59 pm
by sreinhardt
So are you just trying to break it out into the fields you mentioned, no separation of domain from user? Most likely, this needs to be implemented at logstash so that it knows where to place your information. Could you confirm that, and put a copy of one of the existing logs as it shows in the message box on log server?
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 5:06 pm
by FelixForbes
I would like to do some manipulation of the fields (as you say, separate the domain and backslash from the username). At this stage I was just trying to get the message broken out into separate fields.
Here is an entry from the NLS dashboard (the message field)
Code: Select all
2015-01-14 16:39:33,474 [51] INFO [HCF\iqv] Out of CallMQseries
I'd like to split this into $DateTime, $Number, $Loglevel, $UserID, $LogMessage
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 5:20 pm
by tmcdonald
Just to be clear, this is sorta the format you are looking for?
Code: Select all
DateTime = 2015-01-14 16:39:33,474
Number = [51]
Loglevel = INFO
UserID = [HCF\iqv]
LogMessage = Out of CallMQseries
Or for the UserID do you want the [brackets] stripped?
This might take some time to craft up, but the logstash reference docs could be some help in the meantime:
http://logstash.net/docs/1.4.2/filters/grok
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 5:26 pm
by FelixForbes
That is exactly what I am looking for. I would also like to strip out the domain from the username but that is something I can work on for later.
Of all the doco I have read through I have not seen that page on grok yet so I'll have a look through that - thanks for the link.
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 5:38 pm
by tmcdonald
Also take a look at this:
http://grokconstructor.appspot.com/do/match
It lets you put in a string like your log entry and guides you through building a grok pattern to match it. This is what I came up with:
Code: Select all
\A%{TIMESTAMP_ISO8601}%{SPACE}\[[0-9]+\]%{SPACE}%{LOGLEVEL}%{SPACE}\[[a-zA-Z0-9\\]+\]%{SPACE}%{GREEDYDATA}
It will match the following lines:
Code: Select all
2015-01-14 16:39:33,474 [51] INFO [HCF\iqv] Out of CallMQseries
2015-01-14 16:39:33,474 [51] DEBUG [HCF\iqv] Out of CallMQseries
2015-01-14 16:39:33,474 [51] INFO [ABCD\efgh] Out of CallMQseries
2015-01-14 16:39:33,474 [51] INFO [HCF\iqv] Some long message with CAPITALS and numb3rs and punc'tuat"ion
2027-02-30 09:14:00,111 [12345] INFO [HCF\iqv] Out of CallMQseries
Some of it needed to be done by hand, namely the Number and UserID parts, but now all we need to do is tag it. I will need to get back to you on that.
UPDATE:
I got the filter to the following point, cleaned it up a bit:
Code: Select all
grok {
match => [ "message", "\A%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{LOGLEVEL:Loglevel}%{SPACE}\[[a-zA-Z0-9\\]+%{WORD:UserID}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
}
Let me know if this works for you.
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 7:43 pm
by FelixForbes
I've added that filter but I'm not seeing any splitting into fields. I am now getting the tag "_grokparsefailure".
The filter I am using is:
Code: Select all
if [host] == '172.21.5.193' {
grok {
match => [ "message", "\A%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{LOGLEVEL:Loglevel}%{SPACE}\[[a-zA-Z0-9\\]+%{WORD:UserID}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
}
}
Should I be doing this based on host?
Re: Help with filtering a custom log
Posted: Wed Jan 14, 2015 11:26 pm
by FelixForbes
OK after a bit of a break from this I came back with a clear head, spotted a typo and it is now working perfectly - no more _grokparsefailure and everything broken up nicely into fields.
Thank you for taking the time to look at this and provide assistance - I am happy for this to be closed.
Re: Help with filtering a custom log
Posted: Thu Jan 15, 2015 10:24 am
by sreinhardt
Just a slight variation on tmcdonald's post that separates domain and user in a little bit easier way for logstash:
Code: Select all
if [host] == '172.21.5.193' {
grok {
match => [ "message", "%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{WORD:Loglevel}%{SPACE}\[%{WORD:Domain}\\%{USER:Username}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
}
}
While it's only a little less regex, it should make it simpler and quicker for logstash to parse. Definitely not an absolute need to change, just another option.
Re: Help with filtering a custom log
Posted: Thu Jan 15, 2015 1:02 pm
by tmcdonald
I suppose if we wanted to optimize further, we could convert the Number (51 in this case) to an int instead of the default of string:
Code: Select all
match => [ "message", "%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number:int}\]%{SPACE}%{WORD:Loglevel}%{SPACE}\[%{WORD:Domain}\\%{USER:Username}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
This could help in the backend with making it easier to query, since the string "51" is different than the number 51.
But I think we have beaten this horse enough :)
I will close this now.