grok whitespace strip not working

[eloyd]

Why do you have "[\u0000]" in your grok pattern? I would expect to see this:

Code: Select all

grok {
      match => [ "message", "%{TIMESTAMP_ISO8601:logdate} (?<MDNIT>.{8}) (?<V>.{1}) (?<TRANCODE>.{12}) (?<FIELD1>.{2})  (?<FIELD2>.{2})  (?<FIELD3>.{2}) (?<FIELD4>.{6}) (?<PROCESS>.{6})\s+%{INT:APTIME} (?<BKTIME>.{6}) (?<FIELD5>.{6}) (?<CODES>.{8}) (?<FIELD6>.{2}) (?<MC>.{2}) (?<PV>.{2}) (?<PC>.{2}) (?<PX>.{2}) (?<SYS>.{5}) (?<CT>.{2}) (?<RESULTS>.{9}) (?<FIELD7>.{13})"]
    }

[gsl_ops_practice]

Hello,

That pattern was there in an effort to combat the whitespace issue, from a previous round of googling. Removing that pattern does not make a difference, with your suggestion for the APTIME field I am still getting whitespaces in front and consequently unable to graph values over time.

Thanks,
Alex

[eloyd]

Give me some time and let me see if I can craft a proper grok filter for you.

[eloyd]

I may have missed the intention of your field structure, but this works on our system:

Code: Select all

if [type] == "EWtest" {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:logdate} %{INT:MDNIT} %{INT:V} %{WORD:TRANCODE} %{WORD:FIELD1}\s+%{WORD:FIELD2}\s+%{INT:FIELD3}\s+%{INT:FIELD4}\s+%{INT:PROCESS}\s+%{INT:APTIME}\s+%{INT:BKTIME}\s+%{INT:FIELD5}\s+%{WORD:FIELD6}\s+%{INT:MC}\s+%{WORD:PV}\s+%{WORD:PC}\s+%{INT:PX}\s+%{WORD:SYS}\s+%{INT:CT}\s+%{WORD:RESULTS}\s+%{WORD:FIELD7}"]
  }
}

[mcapra]

Using only the grok filter provided by @eloyd above, I am unable to replicate the issue with whitespace using your sample event.

2016_09_09_11_56_42_Dashboard_Nagios_Log_Server.png

[gsl_ops_practice]

Hello,

Apologies reviving this, project is now on the fast track and I'd like to get this working properly.

As per your code I am not seeing any white spaces anymore and it all looks good. Until I try to display those values over time. When I do, I get this error in the GUI:

Code: Select all

ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]

So it looks like the conversion to INT isn't happening properly.

Your input is appreciated.

Thanks,
Alex

[mcapra]

Are you able to share the day's index mappings from when this occurred? Like so if the issue occurred on May 11th:

Code: Select all

curl -XGET 'http://localhost:9200/logstash-2017.05.11/_mapping'

Can you also tell us which values/fields specifically you're referring to?

So it looks like the conversion to INT isn't happening properly.

%{INT} represents a grok pattern, not a field type (not explicitly, anyway). So if I say %{INT:some_field}, then some_field will match the INT grok pattern but not necessarily be stored as an int/integer variable. If you wanted some field to be a specific data type (we'll use long because it's easy) your pattern match in the grok filter would have to look like %{INT:some_field:long} to properly type the field in that instant.

As per your code I am not seeing any white spaces anymore and it all looks good. Until I try to display those values over time. When I do, I get this error in the GUI:

I assume this to mean that you are trying to "Sort By" a specific field in the GUI? Here's an example event:
curl -XGET 'http://localhost:9200/logstash-2017.05. ... rch?size=1'
https://pastebin.com/YV40958z

Code: Select all

{
	"took": 1,
	"timed_out": false,
	"_shards": {
		"total": 5,
		"successful": 5,
		"failed": 0
	},
	"hits": {
		"total": 16801,
		"max_score": 1.0,
		"hits": [{
				"_index": "logstash-2017.05.11",
				"_type": "eventlog",
				"_id": "AVv0zkiDLoUjsjJ7dByf",
				"_score": 1.0,
				"_source": {
					"EventTime": "2017-05-11 01:59:43",
					"Hostname": "WIN-NFRUUIO4D46.DOMAIN.local",
					"Keywords": -9223372036854775808,
					"EventType": "WARNING",
					"SeverityValue": 3,
					"Severity": "WARNING",
					"EventID": 322,
					"SourceName": "Microsoft-Windows-TaskScheduler",
					"ProviderGuid": "{DE7B24EA-73C8-4A09-985D-5BDADCFA9017}",
					"Version": 0,
					"Task": 322,
					"OpcodeValue": 0,
					"RecordNumber": 1208518,
					"ActivityID": "{5D29117E-4827-4F9B-93BB-6CC917ECEB45}",
					"ProcessID": 920,
					"ThreadID": 111444,
					"Channel": "Microsoft-Windows-TaskScheduler/Operational",
					"Domain": "NT AUTHORITY",
					"AccountName": "SYSTEM",
					"UserID": "SYSTEM",
					"AccountType": "User",
					"Category": "Launch request ignored, instance already running",
					"Opcode": "Info",
					"TaskName": "\\test-nrds",
					"TaskInstanceId": "{5D29117E-4827-4F9B-93BB-6CC917ECEB45}",
					"EventReceivedTime": "2017-05-11 01:59:45",
					"SourceModuleName": "eventlog",
					"SourceModuleType": "im_msvistalog",
					"message": "Task Scheduler did not launch task \"\\test-nrds\"  because instance \"{5D29117E-4827-4F9B-93BB-6CC917ECEB45}\"  of the same task is already running.",
					"@version": "1",
					"@timestamp": "2017-05-11T00:00:11.394Z",
					"host": "192.168.67.99",
					"type": "eventlog"
				}
			}
		]
	}
}

Lets focus on the RecordNumber field. Looking at the mapping (think "schema") for the eventlog type, we can see that this field is mapped as a long:
curl -XGET 'http://localhost:9200/logstash-2017.05. ... g/_mapping'
https://pastebin.com/ygFdPLjE (Line 1078)

Code: Select all

"RecordNumber": {
				"type": "long"
			},

And I can consequently sort by this value in the GUI:

2017_05_11_11_38_21_Dashboard_Nagios_Log_Server.png

2017_05_11_11_38_07_Dashboard_Nagios_Log_Server.png

Just a little tidbit, unsure if it's specifically related.

[scottwilkerson]

Hello,

Apologies reviving this, project is now on the fast track and I'd like to get this working properly.

As per your code I am not seeing any white spaces anymore and it all looks good. Until I try to display those values over time. When I do, I get this error in the GUI:

Code: Select all

ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]

So it looks like the conversion to INT isn't happening properly.

Your input is appreciated.

Thanks,
Alex

It is worth pointing out that the field is mapped based on the data going in each day. So if you change the grok filter ann starting when the next day's index is created it may create the field as a long if it believes it is an int, or a string if it believes it is a string.

If you try to run the query that spans across several indexes (multiple days) you may have the same field with 2 different mapping types and this can cause that error.

[gsl_ops_practice]

Hello,

Thank you for your suggestions, I tried them both but not having luck so far:

Step 1 - tried to purge the old indexes to see if the issue is different types of fields across different days - that's not it, same issue

Step 2 - tried to explicitly set the field type to :long, but as per the curl command it is still coming out as string.

Filter:

Code: Select all

%{INT:PROCESSTIME:long}

Curl output:

Code: Select all

,"PROCESSTIME":{"type":"string","norms":{"enabled":false},"fields":{"raw":{"type":"string","index":"not_analyzed","ignore_above":256}}}

[mcapra]

Do you have Elasticsearch logs we can review? Seeing the complete call trace might be helpful in getting a better idea of specifically why that exception is thrown.