Help with filtering a custom log

[FelixForbes]

Hi There,

I have managed to get several logs from Squid proxies and IIS servers coming into my NLS setup nicely and am starting to look at including custom logs. I have one log which produces output similar to the following:

Code: Select all

2015-01-09 12:36:00,051 [13] INFO  [DOMAIN\userid] Closing put queue

I have tried several ways to recognise the fields in the log file. One option I tried was this in my nxlog.conf:

Code: Select all

<Extension w3c>
    Module xm_csv
    Fields $date, $time, $number, $loglevel, $user, $logmessage
    FieldTypes string, string, string, string, string
    Delimiter ' '
</Extension>

<Input MQLog>
    Module    im_file
    File    "D:\\Logs\\MQApplicationLog.log"
    SavePos  TRUE
 
    Exec if $raw_event =~ /^#/ drop();				\
       else							\
       {							\
            w3c->parse_csv();					\
            $SourceName = "MQLog";				\
            $Message = to_json();				\
       }
</Input>

When I look on the console I only see the default fields with all of the data in the message field. Is there something easy that I am missing in the nxlog.conf? I realise the regex is very basic - I am just trying to get something working before I change the field types and remove excess characters etc.

I have also looked at adding a filter via the Global configuration but I wasn't able to get that working either. I have looked through countless nxlog.conf examples but just can't see what I am missing.

[sreinhardt]

So are you just trying to break it out into the fields you mentioned, no separation of domain from user? Most likely, this needs to be implemented at logstash so that it knows where to place your information. Could you confirm that, and put a copy of one of the existing logs as it shows in the message box on log server?

[FelixForbes]

I would like to do some manipulation of the fields (as you say, separate the domain and backslash from the username). At this stage I was just trying to get the message broken out into separate fields.

Here is an entry from the NLS dashboard (the message field)

Code: Select all

2015-01-14 16:39:33,474 [51] INFO  [HCF\iqv] Out of CallMQseries

I'd like to split this into $DateTime, $Number, $Loglevel, $UserID, $LogMessage

[tmcdonald]

Just to be clear, this is sorta the format you are looking for?

Code: Select all

DateTime   = 2015-01-14 16:39:33,474
Number     = [51]
Loglevel   = INFO
UserID     = [HCF\iqv]
LogMessage = Out of CallMQseries

Or for the UserID do you want the [brackets] stripped?

This might take some time to craft up, but the logstash reference docs could be some help in the meantime:

http://logstash.net/docs/1.4.2/filters/grok

[FelixForbes]

That is exactly what I am looking for. I would also like to strip out the domain from the username but that is something I can work on for later.

Of all the doco I have read through I have not seen that page on grok yet so I'll have a look through that - thanks for the link.

[tmcdonald]

Also take a look at this:

http://grokconstructor.appspot.com/do/match

It lets you put in a string like your log entry and guides you through building a grok pattern to match it. This is what I came up with:

Code: Select all

\A%{TIMESTAMP_ISO8601}%{SPACE}\[[0-9]+\]%{SPACE}%{LOGLEVEL}%{SPACE}\[[a-zA-Z0-9\\]+\]%{SPACE}%{GREEDYDATA}

It will match the following lines:

Code: Select all

2015-01-14 16:39:33,474 [51] INFO  [HCF\iqv] Out of CallMQseries
2015-01-14 16:39:33,474 [51] DEBUG  [HCF\iqv] Out of CallMQseries
2015-01-14 16:39:33,474 [51] INFO  [ABCD\efgh] Out of CallMQseries
2015-01-14 16:39:33,474 [51] INFO  [HCF\iqv] Some long message with CAPITALS and numb3rs and punc'tuat"ion
2027-02-30 09:14:00,111 [12345] INFO  [HCF\iqv] Out of CallMQseries

Some of it needed to be done by hand, namely the Number and UserID parts, but now all we need to do is tag it. I will need to get back to you on that.

UPDATE:

I got the filter to the following point, cleaned it up a bit:

Code: Select all

    grok {
        match => [ "message", "\A%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{LOGLEVEL:Loglevel}%{SPACE}\[[a-zA-Z0-9\\]+%{WORD:UserID}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
    }

Let me know if this works for you.

[FelixForbes]

I've added that filter but I'm not seeing any splitting into fields. I am now getting the tag "_grokparsefailure".

The filter I am using is:

Code: Select all

if [host] == '172.21.5.193' {

grok {
        match => [ "message", "\A%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{LOGLEVEL:Loglevel}%{SPACE}\[[a-zA-Z0-9\\]+%{WORD:UserID}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
    }
}

Should I be doing this based on host?

[FelixForbes]

OK after a bit of a break from this I came back with a clear head, spotted a typo and it is now working perfectly - no more _grokparsefailure and everything broken up nicely into fields.

Thank you for taking the time to look at this and provide assistance - I am happy for this to be closed.

[sreinhardt]

Just a slight variation on tmcdonald's post that separates domain and user in a little bit easier way for logstash:

Code: Select all

if [host] == '172.21.5.193' {

grok {
        match => [ "message", "%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number}\]%{SPACE}%{WORD:Loglevel}%{SPACE}\[%{WORD:Domain}\\%{USER:Username}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]
    }
}

While it's only a little less regex, it should make it simpler and quicker for logstash to parse. Definitely not an absolute need to change, just another option.

[tmcdonald]

I suppose if we wanted to optimize further, we could convert the Number (51 in this case) to an int instead of the default of string:

Code: Select all

match => [ "message", "%{TIMESTAMP_ISO8601:DateTime}%{SPACE}\[%{NUMBER:Number:int}\]%{SPACE}%{WORD:Loglevel}%{SPACE}\[%{WORD:Domain}\\%{USER:Username}\]%{SPACE}%{GREEDYDATA:LogMessage}" ]

This could help in the backend with making it easier to query, since the string "51" is different than the number 51.

But I think we have beaten this horse enough :)

I will close this now.