Nagios Support Forum

Support for Nagios products and services
https://support.nagios.com/forum/

can't find reason for _grokparsefailure

https://support.nagios.com/forum/viewtopic.php?t=31629

Page 1 of 1

can't find reason for _grokparsefailure

Posted: Mon Mar 02, 2015 4:56 am
by McCriddle
Hi,

we keep getting a _grokparsefailure on our syslog-messages from our firewall.

We checked our grok-match on http://grokconstructor.appspot.com/do/match#result
In NLS the events get parsed correctly, fields get set but we keep getting "_grokparsefailure".
Maybe someone finds an error we don't see.

thanks in advance

kind reagrds
Mike

example message1:

Code: Select all

<30>2015:03:02-10:49:25 myfw-2 ulogd[26436]: id="2002" severity="info" sys="SecureNet" sub="packetfilter" name="Packet accepted" action="accept" fwrule="163" initf="eth0" outitf="eth1" srcmac="0:50:56:9b:5d:29" dstmac="0:1a:8c:f0:e2:c0" srcip="10.2.111.40" dstip="1.2.3.4" proto="6" length="52" tos="0x00" prec="0x00" ttl="127" srcport="53087" dstport="8000" tcpflags="SYN"
example message2:

Code: Select all

<190>2015:03:02-10:51:24 myfw-2 reverseproxy: id="0299" srcip="1.2.3.4" localip="7.8.9.10" size="0" user="-" host="1.2.3.4" method="GET" statuscode="304" reason="-" extra="-" exceptions="-" time="4272" url="/someurl/images/abcdefg.gif" server="my.vhost.local" referer="https://my.vhost.local/someurl/jsp/bla.jsp" cookie="JSESSIONID=15092056447439F5931BB8648F9C6DAC.EVO1" set-cookie="-"
example parsed message (json):

Code: Select all

{
  "_index": "logstash-2015.03.02",
  "_type": "syslog",
  "_id": "lByV5MV4Q6qNgVyhFcHkkw",
  "_score": null,
  "_source": {
    "message": "<190>2015:03:02-10:51:24 myfw-2 reverseproxy: id=\"0299\" srcip=\"1.2.3.4\" localip=\"7.8.9.10\" size=\"0\" user=\"-\" host=\"1.2.3.4\" method=\"GET\" statuscode=\"304\" reason=\"-\" extra=\"-\" exceptions=\"-\" time=\"16697\" url=\"/someurl/images/icon_print.gif\" server=\"my.vhost.local\" referer=\"https://my.vhost.local/someurl/jsp/abcefg.jsp\" cookie=\"JSESSIONID=15092056447439F5931BB8648F9C6DAC.EVO1\" set-cookie=\"-\"\n",
    "@version": "1",
    "@timestamp": "2015-03-02T09:51:24.000Z",
    "type": "syslog",
    "host": "192.168.132.254",
    "tags": [
      "_grokparsefailure"
    ],
    "priority": 13,
    "severity": 5,
    "facility": 1,
    "facility_label": "user-level",
    "severity_label": "Notice",
    "timestamp": "2015:03:02-10:51:24",
    "fw_node": "myfw-2",
    "fw_sub": "reverseproxy",
    "fw_id": 299,
    "fw_srcip": "1.2.3.4",
    "fw_localip": "7.8.9.10",
    "fw_size": 0,
    "fw_user": "-",
    "fw_host": "1.2.3.4",
    "fw_method": "GET",
    "fw_statuscode": 304,
    "fw_reason": "-",
    "fw_extra": "-",
    "fw_exceptions": "-",
    "fw_time": 16697,
    "fw_ur": "/someurl/images/icon_print.gif",
    "fw_server": "my.vhost.local",
    "fw_referer": "https://my.vhost.local/someurl/jsp/abcdefg.jsp",
    "fw_cookie": "JSESSIONID=15092056447439F5931BB8648F9C6DAC.EVO1",
    "fw_set_cookie": "-"
  },
  "highlight": {
    "message": [
      "<190>2015:03:02-10:51:24 myfw-2 reverseproxy: id=\"0299\" srcip=\"1.2.3.4\" localip=\"7.8.9.10\" size=\"0\" user=\"-\" host=\"1.2.3.4\" method=\"GET\" statuscode=\"304\" reason=\"-\" extra=\"-\" exceptions=\"-\" time=\"16697\" url=\"/someurl/images/icon_print.gif\" server=\"my.vhost.local\" referer=\"https://myvhost.local/someurl/jsp/abcdefg.jsp\" cookie=\"JSESSIONID=@[email protected]@end-highlight@\" set-cookie=\"-\"\n"
    ],
    "fw_cookie": [
      "JSESSIONID=@[email protected]@end-highlight@"
    ]
  },
  "sort": [
    1425289884000,
    1425289884000
  ]
}
filter:

Code: Select all

if [host] == '192.168.132.254' {
	if [type] == 'syslog' {
		grok {
			match => [ 'message', '<190>%{DATA:timestamp} %{DATA:fw_node} %{DATA:fw_sub}: id="%{NUMBER:fw_id:int}" srcip="%{IP:fw_srcip}" localip="%{IP:fw_localip}" size="%{NUMBER:fw_size:int}" user="%{DATA:fw_user}" host="%{IP:fw_host}" method="%{DATA:fw_method}" statuscode="%{NUMBER:fw_statuscode:int}" reason="%{DATA:fw_reason}" extra="%{DATA:fw_extra}" exceptions="%{DATA:fw_exceptions}" time="%{NUMBER:fw_time:int}" url="%{DATA:fw_ur}" server="%{DATA:fw_server}" referer="%{DATA:fw_referer}" cookie="%{DATA:fw_cookie}" set-cookie="%{DATA:fw_set_cookie}"']
			match => [ 'message', '<30>%{DATA:timestamp} %{DATA:fw_node} %{DATA}: id="%{NUMBER:fw_id:int}" severity="%{DATA:fw_severity}" sys="%{DATA:fw_sys}" sub="%{DATA:fw_sub}" name="%{DATA:fw_name}" action="%{DATA:fw_action}" fwrule="%{NUMBER:fw_fwrule:int}" initf="%{DATA:fw_initf}" outitf="%{DATA:fw_outitf}" srcmac="%{DATA:fw_srcmac}" dstmac="%{DATA:fw_dstmac}" srcip="%{DATA:fw_srcip}" dstip="%{DATA:fw_dstip}" proto="%{NUMBER:fw_proto:int}" length="%{NUMBER:fw_length:int}" tos="%{DATA:fw_tos}" prec="%{DATA:fw_prec}" ttl="%{NUMBER:fw_ttl:int}" srcport="%{NUMBER:fw_srcport:int}" dstport="%{NUMBER:fw_dstport:int}" tcpflags="%{DATA:fw_tcpflags}"']
		}
		date {
			match => [ 'timestamp', 'yyyy:MM:dd-HH:mm:ss' ]
		}
	}
}

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 02, 2015 9:08 am
by scottwilkerson
When using the syslog input, it first is trying to match on a grok filter in the syslog input.

When specifying custom filters, either use a different input such as just tcp or udp OR you can just remove your _grokparsefailure tag as it looks like it is parsing all the fields correctly.

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 02, 2015 9:58 am
by McCriddle
hi,

you mean a custom syslog-input with another type like:

syslog {
type => 'mysyslog'
port => 5545
}

or something like

tcp {
type => 'mysyslog'
port => 5545
}


?

thanks in advance

Re: can't find reason for _grokparsefailure

Posted: Tue Mar 03, 2015 9:24 am
by scottwilkerson
Actually in your situation I would just change the filter to this

Code: Select all

if [host] == '192.168.132.254' {
   if [type] == 'syslog' {
      grok {
         match => [ 'message', '<190>%{DATA:timestamp} %{DATA:fw_node} %{DATA:fw_sub}: id="%{NUMBER:fw_id:int}" srcip="%{IP:fw_srcip}" localip="%{IP:fw_localip}" size="%{NUMBER:fw_size:int}" user="%{DATA:fw_user}" host="%{IP:fw_host}" method="%{DATA:fw_method}" statuscode="%{NUMBER:fw_statuscode:int}" reason="%{DATA:fw_reason}" extra="%{DATA:fw_extra}" exceptions="%{DATA:fw_exceptions}" time="%{NUMBER:fw_time:int}" url="%{DATA:fw_ur}" server="%{DATA:fw_server}" referer="%{DATA:fw_referer}" cookie="%{DATA:fw_cookie}" set-cookie="%{DATA:fw_set_cookie}"']
         match => [ 'message', '<30>%{DATA:timestamp} %{DATA:fw_node} %{DATA}: id="%{NUMBER:fw_id:int}" severity="%{DATA:fw_severity}" sys="%{DATA:fw_sys}" sub="%{DATA:fw_sub}" name="%{DATA:fw_name}" action="%{DATA:fw_action}" fwrule="%{NUMBER:fw_fwrule:int}" initf="%{DATA:fw_initf}" outitf="%{DATA:fw_outitf}" srcmac="%{DATA:fw_srcmac}" dstmac="%{DATA:fw_dstmac}" srcip="%{DATA:fw_srcip}" dstip="%{DATA:fw_dstip}" proto="%{NUMBER:fw_proto:int}" length="%{NUMBER:fw_length:int}" tos="%{DATA:fw_tos}" prec="%{DATA:fw_prec}" ttl="%{NUMBER:fw_ttl:int}" srcport="%{NUMBER:fw_srcport:int}" dstport="%{NUMBER:fw_dstport:int}" tcpflags="%{DATA:fw_tcpflags}"']
         remove_tag => ["_grokparsefailure"]
      
      }
      date {
         match => [ 'timestamp', 'yyyy:MM:dd-HH:mm:ss' ]
      }
   }
}
Basically just adding the

Code: Select all

remove_tag => ["_grokparsefailure"]

Re: can't find reason for _grokparsefailure

Posted: Thu Mar 05, 2015 10:08 am
by McCriddle
thanks.



i now tested some configurations.

is it possible to have a match-statement set a field to a constant (that's not in the message, but in the regex)?

e.g.
on first match set a field to "1" on second match to "2"

Code: Select all

match => [ 'message', '<190>%{DATA:timestamp} %{DATA:fw_node} %{DATA:fw_sub}']
match => [ 'message', '<190>%{DATA:timestamp} %{GREEDYDATA}']
then i could add conditional statements

Re: can't find reason for _grokparsefailure

Posted: Fri Mar 06, 2015 4:02 pm
by scottwilkerson
I don't believe this is possible without using multiple grok filters adding the "add_field" for each

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 09, 2015 5:33 am
by McCriddle
thanks a lot,

the hint towards multiple groks got me on the right way.

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 09, 2015 9:24 am
by lgroschen
McCriddle,

Have you gotten this figured out? Feel free to ask more questions, otherwise I'll lock this and you can create a new topic for any other problems you may have.

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 09, 2015 11:08 am
by McCriddle
yep,

i got this one solved.

thanks a lot

Re: can't find reason for _grokparsefailure

Posted: Mon Mar 09, 2015 12:20 pm
by lgroschen
Great, feel free to create a new topic with any other issues/ requests you have.
All times are UTC-05:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited