Add Filter / Field

[aer9480]

Thanks again for your help.

So I've been doing a little more debugging, and I'm still getting the grokparsefailure. I have tried changing the "message" to "syslog_message" because it doesn't have the extra stuff that I don't need in front of the useful data. After that didn't work, I thought that it might be because it has the <14> or another number in there. Could this be what's tripping it up? Could grok be reading the <14> as HTML and not parsing it correctly because of that? Or am I completely off?


Here is my full configuration if that helps:

Code: Select all

# 
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Fri, 22 Apr 2016 10:41:02 -0400
#

#
# Global inputs
#

input {
    tcp {
        port => 5544
        type => syslog
      }
      udp {
        port => 5544
        type => syslog
      }
    tcp {
        type => 'eventlog'
        port => 3515
        codec => json {
            charset => 'CP1252'
        }
    }
    tcp {
        type => 'import_raw'
        tags => 'import_raw'
        port => 2056
    }
    tcp {
        type => 'import_json'
        tags => 'import_json'
        port => 2057
        codec => json
    }
}

#
# Local inputs
#



#
# Global filters
#

filter {
    if [type] == "syslog" {
        grok {
          match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    
        }
         syslog_pri {
        }  
    }
    if [type] == "syslog" {
        grok {
            match => [ "message", "<%{POSINT:SysLogPri}>%{MONTH:Month} %{MONTHDAY:Day} %{TIME:Time} %{GREEDYDATA:FieldThatSaysBit9Server} Bit9 event: text=%{QS:warning} type=%{QS:type} subtype=%{QS:subtype} hostname=%{QS:hostname} username=%{QS:user} date=%{QS:date} ip_address=%{QS:client_ip} process=%{QS:process} file_path=%{QS:path} file_name=%{QS:file} file_hash=%{QS:hash} installer_name=%{QS:installer_exe} policy=%{QS:policy} rule_name=%{QS:rule_name} process_key=%{QS:proc_key} server_version=%{QS:server_ver} file_trust=%{QS:f_trust} file_threat=%{QS:f_threat} process_trust=%{QS:p_trust} process_threat=%{QS:p_threat}" ]
    
        }
    }
}

#
# Local filters
#



#
# Global outputs
#



#
# Local outputs
#

Thanks!
Alan

[hsmith]

I don't think the <14> will be a problem for the latest one that I posted.

Code: Select all

<%{POSINT:SysLogPri}>

As you can see here, I begin it with a <, which it takes as the literal character. Followed by %{POSINT:SysLogPri}, which is going to look for a positive integer after the <, and assign it to the SysLogPri field.

Then we end with >, which should be enough to parse that field just fine. I have no idea why this isn't working, generally if it's working in........I just had an idea.

Open up your filters page, and drag this filter all the way to the top. That way it is the first one that it tries to match.

Apply that configuration and let me know what happens.

[aer9480]

Okay, I just had the idea that it might throw it off, but it wasn't based on much, just a hunch, so thank you for explaining that.

Also, I dragged the filter to the top, save and applied, and when that didn't work, I dragged the other one to the top. (The Apache one was the top one before.) Still didn't work.

Any ideas? Should I have the default Apache one active or inactive? Does it matter?

[hsmith]

The apache one shouldn't matter.

I apologize but I've been in remote sessions literally 75% of my work day. I'll attempt to get this tested for you Monday. Feel free to keep tweaking the filter I created.

Also, if you can PM me a few examples of your logs, it will help because:

a) sometimes logs vary.
b) it will remind me I need to do this Monday

[aer9480]

No worries man, I appreciate your help! I'll PM you some of those logs.

[hsmith]

Just posting here to let everyone know I'm working on this now. I'll edit this post when I have some results.

[hsmith]

I found an extra space in the filter.

It was after Bit9 event:

Try removing one of those spaces and let me know what happens.

[aer9480]

No luck with that. I had actually changed that back before. Is it not giving you the grokparsefailure?

Would you mind posting your full configuration so I can see if I have any other stupid typos or anything?

Thanks!

[hsmith]

Can you please post the contents of your /var/log/logstash/logstash.log file(with any sensitive information obfuscated)?