Need assistance with custom pattern or Filter

[krobertson71]

I have the following application log I am now bringing into Log Server:

Code: Select all

2015/08/18 16:08:13 AgentActionDB                 D   Action HostAccessCheckAddress returned 0
2015/08/18 16:08:13 HttpProtocolHandler           D   Thread 3124: ProcessRequest started on socket 668605920
2015/08/18 16:08:13 AgentCore                     T   Received network incoming connexion (id: 3464) of priority 1 from IP 10.0.103.115 port 45660: handling at once
2015/08/18 16:08:13 AgentCore                     T   Thread 3468: Processing a network incoming connection from 10.0.103.115:45660, id 3464
2015/08/18 16:08:13 AgentCore                     T   Thread 3468: Connection is secured
2015/08/18 16:08:13 AgentCore                     T   Thread 3468: Precision Access Control is requested
2015/08/18 16:08:13 HttpProtocolHandler           D   Thread 3468: Request address is 10.0.103.115 for socket 668559840
2015/08/18 16:08:13 AgentCore                     T   Received network incoming connexion (id: 12016) of priority 1 from IP 10.0.124.113 port 49236: handling at once
2015/08/18 16:08:13 AgentCore                     T   Thread 3912: Processing a network incoming connection from 10.0.124.113:49236, id 12016
2015/08/18 16:08:13 AgentActionDB                 I   Invoke local action HostAccessCheckAddress
2015/08/18 16:08:13 AgentActionDB                 D   Action HostAccessCheckAddress returned 0

As you can see col1 is date col2 is time col3 is the sub-process that is generating the event col4 is the error code (T for Trace I for Info D for Debug ERR for error(not listed in example)).

I am having a hard time breaking this down properly. Basically everything is going into the Message field.

I am not sure if I need to create a pattern file (which I have never done) or use the Filters section in NLS.

Hoping you can give me a quick example of how to best achieve this so I can replicate with other logs.

This source is a Windows Server and this is just a typical single line log file. I was able to configure the NXLog agent to send this file in with the a custom field called 'progam' and name of "AssetCore" so I was thinking I could filter off that, but I am still new to logstash and have other duties that keep me from putting in the time I need.

Here is a screenshot of one of these events as seen in NLS. Any and all help would be Great!!

assetcoreNLS.png

[jdalrymple]

It's rare that you need to add patterns, you need to create yourself a filter.

Something like this should get you started:

Code: Select all

if [program] == 'AssetCore' {
  grok {
    match => [ 'message', '%{DATESTAMP:timestamp} %{WORD:sub_process} *%{WORD:error_code} %{GREEDYDATA:message}' ]
  }
}

[krobertson71]

That worked great! I had some of the syntax wrong before.

But, The fields are broken out properly now, but all the data is still in the message field. What would I add to the filter to remove all the data I am adding to fields and just leave the message there?

assetcoreNLS2.png

Basically I just want this left in the message field: Thread 2972: Request duration: 217 ms, Thread 2972: Request duration: 217 ms.

Does that make sense?

[jdalrymple]

By all rights, the filter I sent should have done what you wanted. I have a hunch it's because I'm doing a match on a field that already exists and we're currently using?

Just to prove me right so that I can then figure out how to mutate message in place can you try this instead:

Code: Select all

if [program] == 'AssetCore' {
  grok {
    match => [ 'message', '%{DATESTAMP:timestamp} %{WORD:sub_process} *%{WORD:error_code} %{GREEDYDATA:egassem}' ]
  }
}

Let's see if that *works* and if it does I'll fiddle with the mutate bit.

[krobertson71]

Think you nailed it on the head. Message is a builtin field I believe.

assetcoreNLS3.png

[jdalrymple]

Disclaimer: I really don't know what I'm doing I'm just doing my best to fill jolson's NLS shoes in his absence.

Try this though:

Code: Select all

if [program] == 'AssetCore' {
  grok {
    match => [ 'message', '%{DATESTAMP:timestamp} %{WORD:sub_process} *%{WORD:error_code} %{GREEDYDATA:message}' ]
    overwrite => [ "message" ]
  }
}

[krobertson71]

That seems to have done it!!

assetcoreNLS4.png