Add Filter / Field

[aer9480]

Hey guys, newbie here. I am trying to make a filter and a field in Nagios using Grok. I have the data coming in from Bit9 into Nagios, but I am having no luck parsing it. This is what my Bit9 alert looks like in the "message" section:

Code: Select all

<14>Apr 20 13:34:39 **BIT9SERVER** Bit9 event:  text="File 'c:\d88f3abb57d62b99a0\setuputility.exe' [ffebe9c434fc6f4d7e8760d3b68f4c0a2dab822c643a0dedee92b83e282704f0] was approved by Publisher 'Microsoft Corporation'." type="Policy Enforcement" subtype="File approved (publisher)" hostname="**Example Hostname**" username="NT AUTHORITY\SYSTEM" date="4/20/2016 1:34:32 PM" ip_address="**IP ADDRESS**" process="c:\windows\softwaredistribution\download\install\ndp46-kb3136000-x64.exe" file_path="c:\d88f3abb57d62b99a0\setuputility.exe" file_name="setuputility.exe" file_hash="ffebe9c434fc6f4d7e8760d3b68f4c0a2dab822c643a0dedee92b83e282704f0" installer_name="bite3fe.tmp" policy="High" rule_name="Approve writes from trusted processes" process_key="0000034d-0000-1a90-01d1-9b095da2eb34" server_version="7.2.1.1562" file_trust="10" file_threat="0" process_trust="10" process_threat="0"

I have the following Grok pattern that parses through the message:

Code: Select all

text=%{QS:warning} type=%{QS:type} subtype=%{QS:subtype} hostname=%{QS:hostname} username=%{QS:user} date=%{QS:date} ip_address=%{QS:client_ip} process=%{QS:process} file_path=%{QS:path} file_name=%{QS:file} file_hash=%{QS:hash} installer_name=%{QS:installer_exe} policy=%{QS:policy} rule_name=%{QS:rule_name} process_key=%{QS:proc_key} server_version=%{QS:server_ver} file_trust=%{QS:f_trust} file_threat=%{QS:f_threat} process_trust=%{QS:p_trust} process_threat=%{QS:p_threat}

It works when I test it on this website: http://grokconstructor.appspot.com/do/match

and it does what I need it to do, but in Nagios, it doesn't output like I want it to. I added the following code into a filter:

Code: Select all

if [type] == "syslog" {
    grok {
        match => [ "message", "text=%{QS:warning} type=%{QS:type} subtype=%{QS:subtype} hostname=%{QS:hostname} username=%{QS:user} date=%{QS:date} ip_address=%{QS:client_ip} process=%{QS:process} file_path=%{QS:path} file_name=%{QS:file} file_hash=%{QS:hash} installer_name=%{QS:installer_exe} policy=%{QS:policy} rule_name=%{QS:rule_name} process_key=%{QS:proc_key} server_version=%{QS:server_ver} file_trust=%{QS:f_trust} file_threat=%{QS:f_threat} process_trust=%{QS:p_trust} process_threat=%{QS:p_threat}" ]

    add_field => [ "warning", "test!" ]

    }
}

But it doesn't add the field "warning" like I told it to do, and it doesn't break the message field down like it does in the online tester website. Any ideas? Thanks!

[hsmith]

Can you show a screenshot of the message on the dashboard? I want to see if the syslog fields are being broken down properly.

[aer9480]

Is this what you're referring to? http://imgur.com/Eh4QLpT

[hsmith]

Yup.

Let's do this...

Go to Administration > Global > Global Configuration

Expand your Output for Syslog (Default), and erase what is in there.

Put this in there instead:

Code: Select all

tcp {
    port => 5544
    type => syslog
  }
  udp {
    port => 5544
    type => syslog
  }

Now, create a new filter that looks like this(on the other side of the screen):

Code: Select all

if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }

    }
     syslog_pri {
    }  
}

Apply that configuration, and let me know if this fixes your problems.

[aer9480]

Hi, thanks for the response. To be honest, I'm not exactly sure what those changes were doing. I tried that, and it added a few more fields to my logs, but when I try to replace things such as

Code: Select all

%{SYSLOGTIMESTAMP:syslog_timestamp}

with things such as

Code: Select all

%{QS:warning}

, it does'nt do what I thought it would do. Should I be editing this filter? Or leaving this one alone and editing my other filter that I made earlier? When I add the new filter that you told me to do, it shows more fields in the summary table, but not any of the fields that I need defined. How can I get around this. Thanks again!

[hsmith]

Should I be editing this filter

Nope! Basically the syslog input in which you replaced breaks down the information in to fields for you, much like you want to do with your filter. The problem with this is that the syslog input expects logs in a certain format. If that doesn't match, there the logs can be dropped, or you can end up with that grokparsefailure. The filter I had you add does what the syslog input is doing. That one should be left alone.

If you can post another screenshot, I can attempt to fix your filter. Another log may be useful as well.

[aer9480]

http://imgur.com/l5VH6Jv

This is the new one, and this is what I want it to look like:

http://i.imgur.com/IZDALAo.jpg

Thanks buddy!

[hsmith]

Does this beast do anything?

Code: Select all

<%{POSINT:SysLogPri}>%{MONTH:Month} %{MONTHDAY:Day} %{TIME:Time} %{GREEDYDATA:FieldThatSaysBit9Server} Bit9 event:  text=%{QS:warning} type=%{QS:type} subtype=%{QS:subtype} hostname=%{QS:hostname} username=%{QS:user} date=%{QS:date} ip_address=%{QS:client_ip} process=%{QS:process} file_path=%{QS:path} file_name=%{QS:file} file_hash=%{QS:hash} installer_name=%{QS:installer_exe} policy=%{QS:policy} rule_name=%{QS:rule_name} process_key=%{QS:proc_key} server_version=%{QS:server_ver} file_trust=%{QS:f_trust} file_threat=%{QS:f_threat} process_trust=%{QS:p_trust} process_threat=%{QS:p_threat}

The beginning is sloppy, but it should match everything.

Keep in mind this will only apply to new logs, not ones you've already received.

[aer9480]

So I'm assuming I added it in wrong, because it gives me this output: http://imgur.com/YCZp51p, which is the same as the last post.

The code I used was:

Code: Select all

if [type] == "syslog" {
    grok {
        match => [ "message", "<%{POSINT:SysLogPri}>%{MONTH:Month} %{MONTHDAY:Day} %{TIME:Time} %{GREEDYDATA:FieldThatSaysBit9Server} Bit9 event:  text=%{QS:warning} type=%{QS:type} subtype=%{QS:subtype} hostname=%{QS:hostname} username=%{QS:user} date=%{QS:date} ip_address=%{QS:client_ip} process=%{QS:process} file_path=%{QS:path} file_name=%{QS:file} file_hash=%{QS:hash} installer_name=%{QS:installer_exe} policy=%{QS:policy} rule_name=%{QS:rule_name} process_key=%{QS:proc_key} server_version=%{QS:server_ver} file_trust=%{QS:f_trust} file_threat=%{QS:f_threat} process_trust=%{QS:p_trust} process_threat=%{QS:p_threat}" ]

    }
}

Does this look like it should be working or am I making a stupid mistake? Thanks!

[hsmith]

It looks right. I can't promise I'll get back to you today, but I'll test it on my system. I can just send a fake copy of your log to my system.