Here's an IIS log filter example for you

[vAJ]

For anyone wanting to get IIS logs filtered, here you are:

Code: Select all

grok {
    match => ["message", "%{DATE} %{TIME} %{IPV4:ip} %{WORD:method} %{URIPATH:stem} %{NOTSPACE:uriquery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:substatus} %{NUMBER:win32status} %{NUMBER:timetaken:int}"]
  }
    mutate {
        replace => [ 'type', 'iis_log' ]
    } 

This is used with NXLog forwarding iis log files to the eventlog input on TCP/3515.

If you're using additional/different W3C fields in your logs, you'll need to adjust the filter.

[tmcdonald]

Thanks for the input! Pun very much intended.

[vAJ]

Ok... so maybe not as graceful as I thought.

This filter wound up taking over my apache logs as well, not sure why... but I figured it was needing a conditional statement wrapping the filter:

Code: Select all

if [SourceModuleName] == 'iis_log1' {

grok {
    match => ["message", "%{DATE} %{TIME} %{IPV4:ip} %{WORD:method} %{URIPATH:stem} %{NOTSPACE:uriquery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:substatus} %{NUMBER:win32status} %{NUMBER:timetaken:int}"]
  }
    mutate {
        replace => [ 'type', 'iis_log' ]
    } 
}

Well, this would up replacing type of ALL events coming in as "iis_log"

So, I'm not sure I know what the flip I'm doing. The grok filter definitely help break out the fields of a W3C log, but it's not complete in the two examples I show here.

Any help would be appreciated.

[lgroschen]

Can you attach your Nxlog config file?

Code: Select all

C:\Program Files (x86)\nxlog\conf\nxlog.conf

One thing to note is that when you add a <input filename> all the events you are going to get in log server will be named filename when the logs are sent. If you are monitoring logs on the Windows client you will want a regular event log input and an additional input for iis_log. Then I think your iis specific filter will only find the correct logs and change the type.


/Luke

[vAJ]

Code: Select all

define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert
 
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
 
# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
#    Module xm_fileop
#</Extension>
 
<Extension json>
    Module      xm_json
</Extension>
 
<Extension syslog>
    Module xm_syslog
</Extension>
 
<Input internal>
    Module im_internal
</Input>
 
# Watch your own files
#<Input file1>
#    Module   im_file
#    File     '%ROOT%\data\nxlog.log'
#    SavePos  TRUE
#</Input>

# Watch IIS log files
<Input iis_log1>
    Module   im_file
    File     'D:\inetpub\logs\LogFiles\W3SVC2\u_ex*.log'
    SavePos  TRUE
    Exec     $Message = $raw_event;
</Input>

# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
 
# Uncomment im_mseventlog for Windows XP/2000/2003
#   Module im_mseventlog
</Input>

<Output out>
    Module om_tcp
    Host 10.80.105.50
    Port 3515
	
    Exec  $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
    Exec  $raw_event = to_json();
	
	# Uncomment for debug output
	# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>
 
<Route 1>
    Path internal, iis_log1, eventlog => out
</Route>

I named the input for the IIS logs as "iis_log1". But I figured only those IIS logs would be tagged with that.

When I had the last filter in place, it was changing the type of all events, from all hosts to "iis_log" . The first filter was changing just apache and IIS logs to "iis_log"

[lgroschen]

Can you also post your input and filters that are currently applied to your Log Server? The nxlog.conf looks fine.

When you say the filter is changing the apache log type do you mean it is changing apache logs coming from nxlog or apache coming from an rsyslog client?

[vAJ]

top of post was my first filter. After applying that, it was marking all apache from rsyslog and IIS from nxlog as type==iis_log

Next filter down marked all events from all hosts as type==iis_log

Aside from that, I just have the default apache filter applied:

Code: Select all

if [program] == 'apache_access' {
    grok {
        match => [ 'message', '%{COMBINEDAPACHELOG}']
    }
    date {
        match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
    }
    mutate {
        replace => [ 'type', 'apache_access' ]
         convert => [ 'bytes', 'integer' ]
         convert => [ 'response', 'integer' ]
    }
}
 
if [program] == 'apache_error' {
    grok {
        match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
    }
    mutate {
        replace => [ 'type', 'apache_error' ]
    }
}

[lgroschen]

Here is the message filter I found for IIS in some logstash examples. Try adding this in a new filter and then mutate the type to IIS through the filter.

Add something similar to this and see if it only mutates IIS logs based on the match it makes to the message (you may have to check syntax for [program] like you have it in your other example as [SourceModuleName]):

Code: Select all

if [program] == 'iis_log1' {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:logtime} %{IP:host_ip} %{URIPROTO:method} %{URIPATH:path} (?:-|%{NOTSPACE:uri_query}sern) %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
    } 
    mutate {
        replace => [ 'type', 'iis_log' ]
     }
}

[vAJ]

That isn't matching anything. Neither when I set

Code: Select all

if [SourceModuleName] == 'iis_log1'

[abrist]

The devs are spinning up an IIS server to test this. More info soon.