Syslog severities not parsed

[WillemDH]

Hello,

I'm a litle bit stuck on another quite complex Logstash filter I'm working on. The Logstash filter is for our Infoblox device. Untill recently I used an input like this:

Code: Select all

syslog {
    type => 'syslog-infoblox'
    port => 5545
}

But the message field was not getting parsed like we wanted it, and as the default syslog filter was used, the filters I tried using on top of that did not seem to get parsed. So I changed the input to:

Code: Select all

tcp {
    type => 'syslog-infoblox'
    port => 5545
}

And started working on a custom filter from scratch.

At the moment I have this:

Code: Select all

if [type] == "syslog-infoblox" {
    grok {     
      break_on_match => false
      match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{HOSTNAME} %{SYSLOGPROG}\: %{GREEDYDATA:info}" ]
      add_tag => "grokked_syslog_infoblox"      
    } 
}

Followed by:

Code: Select all

if [program] == "dhcpd" {
    grok {          
        patterns_dir => "/usr/local/nagioslogserver/logstash/patterns"
        match => [ "info", "%{DHCPREQUEST:dhcprequest} %{GREEDYDATA:info}" ]   
        match => [ "info", "%{DHCPMAP:dhcpmap}(: | )%{GREEDYDATA:info}" ]
        match => [ "info", "%{DHCPOTHER:dhcpother}(: | )%{GREEDYDATA:info}" ]           
        remove_tag => "grokked_syslog_infoblox"
        add_tag => "grokked_syslog_infoblox_dhcpd"
        overwrite => [ "info" ]
    }
    grok {    
        patterns_dir => "/usr/local/nagioslogserver/logstash/patterns" 
        match => [ "info", "on %{GREEDYDATA:dhcpon} to %{GREEDYDATA:dhcptoipv6} \(%{GREEDYDATA:dhcptohostname}\) via %{GREEDYDATA:dhcpvia} relay %{GREEDYDATA:dhcprelay} lease-duration %{GREEDYDATA:dhcpleaseduration} offered-duration %{GREEDYDATA:dhcpofferedduration} uid %{GREEDYDATA:dhcpuid}" ]
        match => [ "info", "on %{GREEDYDATA:dhcpon} to %{GREEDYDATA:dhcptoipv6} \(%{GREEDYDATA:dhcptohostname}\) via %{GREEDYDATA:dhcpvia} relay %{GREEDYDATA:dhcprelay} lease-duration %{GREEDYDATA:dhcpleaseduration} uid %{GREEDYDATA:dhcpuid}" ]
        match => [ "info", "on %{GREEDYDATA:dhcpon} to%{GREEDYDATA:dhcptoipv6} \(%{GREEDYDATA:dhcptohostname}\) via %{GREEDYDATA:dhcpvia} relay %{GREEDYDATA:dhcprelay} lease-duration %{GREEDYDATA:dhcpleaseduration}" ]
        match => [ "info", "for %{GREEDYDATA:dhcpfor} from %{GREEDYDATA:dhcpfrom} via %{GREEDYDATA:dhcpvia} uid %{GREEDYDATA:dhcpuid}" ]   
        match => [ "info", "from %{GREEDYDATA:dhcpfrom} via %{GREEDYDATA:dhcpvia} uid %{GREEDYDATA:dhcpuid}" ]    
        match => [ "info", "from %{GREEDYDATA:dhcpfrom} via %{GREEDYDATA:dhcpvia}" ]  
        match => [ "info", "to %{IP:dhcptoip} \(%{GREEDYDATA:dhcptoipv6}\) via %{GREEDYDATA:dhcpvia}" ]  
        match => [ "info", "%{GREEDYDATA:info}" ]  
        overwrite => [ "info" ]     
        remove_tag => "grokked_syslog_infoblox_dhcpd"
        add_tag => "grokked_syslog_infoblox_dhcpd_rest"
    }
}

The results is very nice. All ipv4 and v6 addresses are captured fine, but there is one thing that is missing after the two above filters are applied and that is the severity_label. When I used the syslog input, the severity_label was somehow retrieved from the SYSLOG5424PRI pattern. There is no other hint in the Infoblox syslog messages to get the severity_label. Some research showed that it is possible to get the severity_label from the SYSLOG5424PRI field.

Code: Select all

             emergency   alert   critical   error   warning   notice   info   debug
 kernel              0       1          2       3         4        5      6       7
 user                8       9         10      11        12       13     14      15
 mail               16      17         18      19        20       21     22      23
 system             24      25         26      27        28       29     30      31
 security           32      33         34      35        36       37     38      39
 syslog             40      41         42      43        44       45     46      47
 lpd                48      49         50      51        52       53     54      55
 nntp               56      57         58      59        60       61     62      63
 uucp               64      65         66      67        68       69     70      71
 time               72      73         74      75        76       77     78      79
 security           80      81         82      83        84       85     86      87
 ftpd               88      89         90      91        92       93     94      95
 ntpd               96      97         98      99       100      101    102     103
 logaudit          104     105        106     107       108      109    110     111
 logalert          112     113        114     115       116      117    118     119
 clock             120     121        122     123       124      125    126     127
 local0            128     129        130     131       132      133    134     135
 local1            136     137        138     139       140      141    142     143
 local2            144     145        146     147       148      149    150     151
 local3            152     153        154     155       156      157    158     159
 local4            160     161        162     163       164      165    166     167
 local5            168     169        170     171       172      173    174     175
 local6            176     177        178     179       180      181    182     183
 local7            184     185        186     187       188      189    190     191

An example message that should have a informational severity_label =>

Code: Select all

<134>Jul 15 08:53:58 10.23.17.250 dhcpd[10639]: DHCPACK to 10.23.136.206 (f8:b1:56:de:be:4d) via eth2

And another that should have an error severity_label =>

Code: Select all

<131>Jul 15 08:53:58 10.15.17.251 dhcpd[31808]: Unable to add forward map from PRGEN11420.gep.gt.be to 10.15.129.164: REFUSED

As you can see apart from the SYSLOG5424PRI there is no other way to get the severity_label.
So how can I edit my above filters so Logstash retrieves the correct severity_label from the SYSLOG5424PRI field?

EDIT: I'm using these custom Logstash patterns:

Code: Select all

DHCPREQUEST (DHCPACK|DHCPREQUEST|DHCPINFORM|DHCPOFFER|DHCPDISCOVER|DHCPDECLINE|DHCPRELEASE|DHCPNAK|DHCPEXPIRE|RELEASE)
DHCPMAP (Added new forward map|Removed forward map|Unable to add forward map|Added reverse map|Removed reverse map|Addition of reverse map|Addition of forward map|Attempt to add forward map|Unable to add reverse map)
DHCPOTHER (DDNS|Dynamic DNS|icmp_echorequest)

Grtz

Willem

[jolson]

This cannot be done using grok filters alone - the syslog input, in addition to parsing using a grok filter, will apply another filter: https://www.elastic.co/guide/en/logstas ... g_pri.html

Look into the above filter, I believe it will do what you want it to.

Filter plugin for logstash to parse the PRI field from the front of a Syslog (RFC3164) message. If no priority is set, it will default to 13 (per RFC).

This filter is based on the original syslog.rb code shipped with logstash.

[WillemDH]

Jesse,

I tried usin this syslog_pri plugin, but I'm afraid I don't really understand how to use it. I can't find any practical examples to get the severity_label from the syslog5424_pri field. I tried with this filter:

Code: Select all

if [type] == "syslog-infoblox" {
    grok {     
        break_on_match => false
        match => [ "message", "\A%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP} %{HOSTNAME} %{SYSLOGPROG}\: %{GREEDYDATA:info}" ]
        add_tag => "grokked_syslog_infoblox"      
    }
    syslog_pri {
        syslog_pri_field_name = syslog5424_pri
    }
}

But it doesn't really seems to do anything. Any suggestion?

EDIT: I made a post on the Elastic forum about this : https://discuss.elastic.co/t/severity-l ... nput/25720

and I'm asked to create an output: As I did not use the output part in the global configuration, i'm not 100 % sure what it does.

Code: Select all

if [type] == "syslog-infoblox" { 
    codec => rubydebug 
}

Would the above output work for me? What does it do?

Grtz

Willem

[jolson]

Willem,

This should be working the way that you have it set up - the syslog_pri filter should be able to see the field you have specified (syslog5424_pri). The only thing I can think of that may be going wrong here is that your syntax looks a little off (missing quotes). Try the following:

Code: Select all

syslog_pri {
    syslog_pri_field_name = "syslog5424_pri"
}

Does that work for you?

Would the above output work for me? What does it do?

The output specified on the elasticsearch forum will output all of your inbound log data to standard out on your terminal.

[WillemDH]

Could you explain me how I can read this output? I ssh to NLS node and then?

[jolson]

To read the output on a node, perform the following procedure as root:

1. Add 'stdout' to your logstash output. I am doing this manually.

Code: Select all

cat /usr/local/nagioslogserver/logstash/etc/conf.d/999_outputs.conf
#
# Logstash Configuration File
# Dynamically created by Nagios Log Server
#
# DO NOT EDIT THIS FILE. IT WILL BE OVERWRITTEN.
#
# Created Mon, 29 Jun 2015 16:48:24 -0400
#

#
# Required output for Nagios Log Server
#

output {
stdout { codec => rubydebug }
    elasticsearch {
        cluster => 'xxxxxxxxxxxx'
        host => 'localhost'
        document_type => '%{type}'
        node_name => 'xxxxxxxxxx'
        protocol => 'transport'
        workers => 4
    }
}

2. Stop the logstash daemon.

Code: Select all

service logstash stop

3. Start the logstash daemon manually.

Code: Select all

/usr/local/nagioslogserver/logstash/bin/logstash agent -f /usr/local/nagioslogserver/logstash/etc/conf.d

Once started, you will see all of your inbound logs on the screen of your terminal.