Input type is getting Grokked even when the filter is off

[cpatterson1]

We have an input named IIS_Requests that monitor our IIS logs. We noticed an issue where if the IIS logs were perfect configured they quit being displayed in Nagios altogether. However, if there was an issue (such as having too few fields in the log file) which caused the Grok to fail, then they would show up in the interface. After a bunch of testing, we turned off the Grok filter we are using for IIS requests and all the IIS log files began pouring in. The strange part, however, is with that filter disabled not only were the requests coming in, but they were fully Grokked.

I then pulled the entire Filter file, and there is nothing there hidden in a different Filter that would be causing these IIS_Requests to Grok (let alone correctly). To further test this theory I turned off all Filters. Everything came in as a pure message dump, except the IIS logs which were still being Grokked. It seems like it is very likely that this is occurring somewhere else in the systems.

We have a 5-node cluster, and we are running 1.4.1. This was working at one point, and we are wondering if the latest update caused this issue. Any assistance is greatly appreciated!

[cpatterson1]

Below are the Input and Filter we are using (based on this: https://exchange.nagios.org/directory/A ... rd/details)

Here is the Input:

Code: Select all

tcp {
    type => 'IIS_requests'
    tags => 'IIS_requests'
    port => 5142
    codec => json
}

And here is the filter:

Code: Select all

if [type] == 'IIS_requests' {
  grok {
    match => ['message', '%{DATESTAMP:timestamp} %{DATA:Host} %{DATA:hostip} %{WORD:method} %{DATA:request} (?:%{NOTSPACE:param}|-) %{NUMBER:port} (?:%{DATA:username}|-) %{DATA:clientip} (?:%{NOTSPACE:agent}|-) (?:%{SPACE}|-|%{DATA:referer}) %{DATA:Source_IP} %{NUMBER:response} %{NUMBER:status} %{NUMBER:sub_status} %{NUMBER:time_taken}']   
}
}

[hsmith]

Can you show me both the input, and the filter you are using?

[cpatterson1]

They should be posted just above your comment (thought to do that after I submitted).

[hsmith]

Can you see anything in /var/log/logstash/logstash.log when these messages are getting dropped?

[cpatterson1]

Sorry for the delay. Obviously I see a lot coming through in the logs, but I do not see any of the type IIS_Requests in that log.

[hsmith]

If everything is working correctly, logstash.log should be very, very quiet. See if you can notice any error messages in it.

[cpatterson1]

Again, sorry for the slow response. I pulled a snippet of what is going through the log and I am not necessarily seeing anything indicating errors within Logstash.log (the only thing containing "errors" are the logs it is pulling in that are errors on the remote devices). But it is far from quiet. The current log on one of our servers is over 5.2 GB.

I found a specific logfile that was dumped into this stash, and it is making it into the front end. So it doesn't seem like what we are seeing in this stash is failing to make it into the actual front end portion to be searchable.

[hsmith]

Something is definitely wrong. Here's how big that file is on my system:

Code: Select all

[root@log ~]# ls -lh /var/log/logstash/logstash.log
-rw-r--r-- 1 nagios users 352K Jun  8 11:21 /var/log/logstash/logstash.log

And mine is super big, but I've been doing a lot of stuff today that will produce noisy error logs.

One thing you can try is switching to using raw TCP/UDP for your inputs instead of the syslog input.

Go to Administration > Global > Global Configuration and expand the syslog input.

Replace what is there with this:

Code: Select all

tcp {
    port => 5544
    type => syslog
  }
  udp {
    port => 5544
    type => syslog
  }
  

This will stop Logstash from dropping logs with a syslog format that it doesn't like, which is the default behavior.

Your syslogs will no longer be broken down in to nice fields, because we're not using the syslog input anymore, but we can have that process done with this grok filter:

Code: Select all

  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }

Give this a shot.

[cpatterson1]

Alright, I put this change in place for all of the syslog types we had. This appears to have fixed the log issue as I have not seen a log come in for about 10-15 minutes to that file now and we're getting those log types. So, it appears we're not getting any errors at this point, at least not at a level that would indicate this issue at this point.