Page 1 of 2
Here's an IIS log filter example for you
Posted: Thu Oct 30, 2014 11:03 am
by vAJ
For anyone wanting to get IIS logs filtered, here you are:
Code: Select all
grok {
match => ["message", "%{DATE} %{TIME} %{IPV4:ip} %{WORD:method} %{URIPATH:stem} %{NOTSPACE:uriquery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:substatus} %{NUMBER:win32status} %{NUMBER:timetaken:int}"]
}
mutate {
replace => [ 'type', 'iis_log' ]
}
This is used with NXLog forwarding iis log files to the
eventlog input on TCP/3515.
If you're using additional/different W3C fields in your logs, you'll need to adjust the filter.
Re: Here's an IIS log filter example for you
Posted: Thu Oct 30, 2014 11:06 am
by tmcdonald
Thanks for the input! Pun very much intended.
Re: Here's an IIS log filter example for you
Posted: Thu Oct 30, 2014 3:03 pm
by vAJ
Ok... so maybe not as graceful as I thought.
This filter wound up taking over my apache logs as well, not sure why... but I figured it was needing a conditional statement wrapping the filter:
Code: Select all
if [SourceModuleName] == 'iis_log1' {
grok {
match => ["message", "%{DATE} %{TIME} %{IPV4:ip} %{WORD:method} %{URIPATH:stem} %{NOTSPACE:uriquery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referer} %{NUMBER:status} %{NUMBER:substatus} %{NUMBER:win32status} %{NUMBER:timetaken:int}"]
}
mutate {
replace => [ 'type', 'iis_log' ]
}
}
Well, this would up replacing type of ALL events coming in as "iis_log"
So, I'm not sure I know what the flip I'm doing. The grok filter definitely help break out the fields of a W3C log, but it's not complete in the two examples I show here.
Any help would be appreciated.
Re: Here's an IIS log filter example for you
Posted: Thu Oct 30, 2014 3:48 pm
by lgroschen
Can you attach your Nxlog config file?
Code: Select all
C:\Program Files (x86)\nxlog\conf\nxlog.conf
One thing to note is that when you add a <input
filename> all the events you are going to get in log server will be named
filename when the logs are sent. If you are monitoring logs on the Windows client you will want a regular event log input and an additional input for iis_log. Then I think your iis specific filter will only find the correct logs and change the type.
/Luke
Re: Here's an IIS log filter example for you
Posted: Fri Oct 31, 2014 9:31 am
by vAJ
Code: Select all
define ROOT C:\Program Files (x86)\nxlog
define CERT %ROOT%\cert
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
# Include fileop while debugging, also enable in the output module below
#<Extension fileop>
# Module xm_fileop
#</Extension>
<Extension json>
Module xm_json
</Extension>
<Extension syslog>
Module xm_syslog
</Extension>
<Input internal>
Module im_internal
</Input>
# Watch your own files
#<Input file1>
# Module im_file
# File '%ROOT%\data\nxlog.log'
# SavePos TRUE
#</Input>
# Watch IIS log files
<Input iis_log1>
Module im_file
File 'D:\inetpub\logs\LogFiles\W3SVC2\u_ex*.log'
SavePos TRUE
Exec $Message = $raw_event;
</Input>
# Windows Event Log
<Input eventlog>
# Uncomment im_msvistalog for Windows Vista/2008 and later
Module im_msvistalog
# Uncomment im_mseventlog for Windows XP/2000/2003
# Module im_mseventlog
</Input>
<Output out>
Module om_tcp
Host 10.80.105.50
Port 3515
Exec $tmpmessage = $Message; delete($Message); rename_field("tmpmessage","message");
Exec $raw_event = to_json();
# Uncomment for debug output
# Exec file_write('%ROOT%\data\nxlog_output.log', $raw_event + "\n");
</Output>
<Route 1>
Path internal, iis_log1, eventlog => out
</Route>
I named the input for the IIS logs as "iis_log1". But I figured only those IIS logs would be tagged with that.
When I had the last filter in place, it was changing the type of all events, from all hosts to "iis_log" . The first filter was changing just apache and IIS logs to "iis_log"
Re: Here's an IIS log filter example for you
Posted: Fri Oct 31, 2014 10:42 am
by lgroschen
Can you also post your input and filters that are currently applied to your Log Server? The nxlog.conf looks fine.
When you say the filter is changing the apache log type do you mean it is changing apache logs coming from nxlog or apache coming from an rsyslog client?
Re: Here's an IIS log filter example for you
Posted: Fri Oct 31, 2014 10:55 am
by vAJ
top of post was my first filter. After applying that, it was marking all apache from rsyslog and IIS from nxlog as type==iis_log
Next filter down marked all events from all hosts as type==iis_log
Aside from that, I just have the default apache filter applied:
Code: Select all
if [program] == 'apache_access' {
grok {
match => [ 'message', '%{COMBINEDAPACHELOG}']
}
date {
match => [ 'timestamp', 'dd/MMM/yyyy:HH:mm:ss Z' ]
}
mutate {
replace => [ 'type', 'apache_access' ]
convert => [ 'bytes', 'integer' ]
convert => [ 'response', 'integer' ]
}
}
if [program] == 'apache_error' {
grok {
match => [ 'message', '\[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[%{WORD:class}\] \[%{WORD:originator} %{IP:clientip}\] %{GREEDYDATA:errmsg}']
}
mutate {
replace => [ 'type', 'apache_error' ]
}
}
Re: Here's an IIS log filter example for you
Posted: Fri Oct 31, 2014 11:31 am
by lgroschen
Here is the message filter I found for IIS in some logstash examples. Try adding this in a new filter and then mutate the type to IIS through the filter.
Add something similar to this and see if it only mutates IIS logs based on the match it makes to the message (you may have to check syntax for [program] like you have it in your other example as [SourceModuleName]):
Code: Select all
if [program] == 'iis_log1' {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:logtime} %{IP:host_ip} %{URIPROTO:method} %{URIPATH:path} (?:-|%{NOTSPACE:uri_query}sern) %{NUMBER:port} %{NOTSPACE:username} %{IP:client_ip} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
}
mutate {
replace => [ 'type', 'iis_log' ]
}
}
Re: Here's an IIS log filter example for you
Posted: Fri Oct 31, 2014 3:08 pm
by vAJ
That isn't matching anything. Neither when I set
Code: Select all
if [SourceModuleName] == 'iis_log1'
Re: Here's an IIS log filter example for you
Posted: Mon Nov 03, 2014 3:58 pm
by abrist
The devs are spinning up an IIS server to test this. More info soon.