Forwarding multiple logs from central syslog

[vAJ]

Ok guys. Here's a curve.

We utilize a central syslog infrastructure in our environment. I have both Windows log forwarders and network devices pumping log data to Kiwi Syslog. I then filter and forward as needed. All go to SEIM, some are presented in the console views, some or all get forwarded to the various log analysis tools I've demo'd.

Below is the raw data from a syslog message from a network device. The whole original log message comes in as the "message" field from the forwarding server. Any ideas on how to break this out? Is this done through filters?

Code: Select all

{
  "_index": "logstash-2014.10.21",
  "_type": "syslog",
  "_id": "CA7X_HsHQ8yca-OmIL1Cow",
  "_score": null,
  "_source": {
    "message": "<43>Oct 21 14:02:23 10.40.4.245 Kiwi_Syslog_Server Original Address=10.40.4.245 WLC-101A.UNI1: *dot1xMsgTask: Oct 21 19:02:16.510: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:3062 Max EAP identity request retries (3) exceeded for client 74:f0:6d:51:f3:01",
    "@version": "1",
    "@timestamp": "2014-10-21T23:02:19.577Z",
    "type": "syslog",
    "host": "10.70.5.24",
    "tags": [
      "_grokparsefailure"
    ],
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice"
  },
  "sort": [
    1413932539577,
    1413932539577
  ]
}

[tmcdonald]

This would be done through filters, correct. I haven't gotten to write too many of them, but I did just review some documentation that should be released shortly.

Under Administration -> Global Configuration, you will want to add a filter. From the doc:

The structure of
the filters are in JSON format and look like this:

Code: Select all

filter {
    <filter_type> {
        <filter_action> => [ “<selected_field>”, “<selected_pattern>” ]
    }
}

<filter_type> is the type of filter plugin that will be used to match the field and pattern you are looking for. Logstash allows a large
amount of possible filter types, but we will explore some that are more useful for manipulating logs. For our example we will be using
grok to do our filtering. To use any of the possible filter plugins you can import them from the Logstash website. The filters section
column on the Logstash docs page would be where plugins like date, grep, grok and elasticsearch are located. Mutate will also be
useful for changing any field in the logs we receive so you can customize logs as much as you want.
<filter_action> is the action that will be taken using the filter type. The most common using our grok plugin are match,
add_field, add_tag. Here is an explanation and more details about grok from Logstash: http://Logstash.net/docs/1.4.2/filters/grok.

The next section will likely be rewritten to reflect more accurately what is in NLS by default. The example is for PHP and mysqld, but the general concept should work for anything.

Grok Filter
Grok is a plugin that is used by Logstash for making specific filters using regex and matching. The plugin allows a lot of customization
and will be used heavily in making custom filters in your Nagios Log Server configuration.
<matching_field> will be the log field that we are attempting to match. You can choose things from 'hostname' to 'message' and
customize whatever is needed.
<matching_pattern> relates to the pattern that will be looked for when the field is given. For example the field 'message' will be
populated with whatever log message was included. For PHP a log message may look like the following example:

Code: Select all

[Thu Oct 02 16:05:21 2014] [error] [client 127.0.0.1] PHP Notice: Undefined variable: _SESSION
in /path/to/file on line 202, referer: http://<address>/page/request/

This message from the apache error_log is showing a PHP Notice that there is an undefined variable. It shows the time the notice was
posted, the type which is [error] here, the client sending the log posting, the PHP log type and message, the variable and path to the
offending file, the line number the offending code is on and the referring page that generated the request that resulted in the notice.
This is a lot of information to take advantage of and this specific PHP notice isn't even as large as many of them could potentially be.
This is what will be present in the message field. To take control of this message and filter the whole message or parts of it you will use
a grok pattern that relates to the message.

Here is a grok filter example searching for any logs with the mysqld program name. It will match the message with the following
matching in the message field:

Code: Select all

filter {
    if [program] == “mysqld” {
        grok {
            match => [ “message”, “^%{NUMBER:date} *%{NOTSPACE:time} ” ]
        }
        mutate {
            replace => [ "type", "mysql_log" ]
        }
    }
}

This filter is designed to match mysqld message contents and replace the log type with mysql_log.
Mutate, as mentioned before, is another very useful plugin used in Logstash filtering. This allows you to replace or append a value in
any field with a new value that may replace the whole field or add, delete or append portions of it. In our example above we are
changing the log type which would be 'syslog' and replace it with 'mysql_log' so that we can differentiate between our normal syslogs
and our mysql syslogs. It knows which syslogs to mutate because the [program] must be mysqld and must match the grok filter
message pattern.

Do you have an example of the raw format of the log message? Or does it actually come in as JSON? Either way, it's mostly regex that does the breaking into fields, and we should be able to work out a filter for you.

[vAJ]

Code: Select all

<43>Oct 21 14:02:23 10.40.4.245 Kiwi_Syslog_Server Original Address=10.40.4.245 WLC-101A.UNI1: *dot1xMsgTask: Oct 21 19:02:16.510: %DOT1X-3-MAX_EAP_RETRIES: 1x_auth_pae.c:3062 Max EAP identity request retries (3) exceeded for client 74:f0:6d:51:f3:01

This is the raw message that the Syslog daemon receives from the network device. Not sure what the <43> tag is, though.

[scottwilkerson]

This guy's site is going to be your friend when creating grok patterns
https://grokdebug.herokuapp.com/

Throw you log line in the Input
Put something like the following in the Pattern

Code: Select all

<%{NONNEGINT:facility}>%{SYSLOGTIMESTAMP:timestamp} %{IP:host} %{WORD:syslogserver} %{GREEDYDATA:message}

Check the box "Named Captures Only"
And below you will see how it will be split up

The syslog input you are using expects RFC3164 syslog messages, you may want to setup a new TCP input that has a different pattern match.

The filter above would look like

Code: Select all

grok {
        match => [ 'message', '<%{NONNEGINT:facility}>%{SYSLOGTIMESTAMP:timestamp} %{IP:host} %{WORD:syslogserver} %{GREEDYDATA:message}']
    }

[vAJ]

Thanks Scott. I can enable RFC3164 header on the forwarder. I'll see how that changes the messages.

All great info. Again, thanks!

Very excited about this product launch.

-AJ

[sreinhardt]

So are we! As always let us know if you have more questions!