Unexpected results escaping characters in search queries

[mliverez]

Searching a field for a string with a special character is not filtering the results to include the special character.
This is where I am entering the query

Search_Query.png

I am expecting to get back all entries where the message field contains the text assignmentTotal:- instead it is returning all entries where the message filed contains the text assignmentTotal ignoring the :- portion

Here is an entry that is returned correctly

Should_Be_Included.png

The results though also include this entry

Should_Not_Be_Included.png

I have tried many different variations to get this query to work but I cannot get the results I need.

Tested searches

Following queries return all entries with assignmentTotal ignoring :-
message:"assignmentTotal\:\-"
message:assignmentTotal\:-
message:assignmentTotal\:-
message:assignmentTotal\:\-
message:"assignmentTotal:-"
message:"assignmentTotal?\-"
message:"assignmentTotal\*\-"
message:"assignmentTotal\?\-"
message:"assignmentTotal*\-"
message:"assignmentTotal?-"
message:"assignmentTotal*-"
message:"assignmentTotal\?-"
message:"assignmentTotal\*-"
message:"assignmentTotal\?\-"
message:"assignmentTotal\*\-"
message:"assignmentTotal\?-"
message:"assignmentTotal\*-"
message:assignmentTotal\?\-
message:assignmentTotal\*\-
message:assignmentTotal\?-
message:assignmentTotal\*-

Following queries return no results
"message:assignmentTotal\:\-"
'message:assignmentTotal\:\-'
"message:assignmentTotal:-"

Following query returns QueryParsingException[[logstash-2018.01.27] Failed to parse query [message:assignmentTotal:-]]
message:assignmentTotal:-

Following query returns QueryParsingException[[logstash-2018.02.26] Failed to parse query [message:assignmentTotal:\-]]
message:assignmentTotal:\-

If anyone has any insight into what I am doing wrong I would appreciate the help.

[cdienger]

The :and -are not searchable due to the standard analyzer used to tokenize the data: https://www.elastic.co/guide/en/elastic ... lyzer.html . A filter for or excluding other unique strings would be needed to only get the desired data.

In theory the analyzer is configurable on the elasticsearch end but unadvised as NLS is written with the standard analyzer in mind. I would be happy to file a feature request for something like this.

[mliverez]

Thanks @cdienger, I was afraid that was the case, but since the documentation that the query form links to for LUCENE query string syntax states that these characters could be used if escaped, see Reserved Characters Section https://www.elastic.co/guide/en/elastic ... characters
I was expecting it to work and thought maybe I was just doing it incorrectly since the only documentation offered stated it should work. If anyone has any ideas on how I can filter my results for all places where my field contains negative numbers I would appreciate they help since there is no other field that separates the negative entries from the positive ones. I am adding on a log entry specifying the word negative to aid filtering future logs but I would still like to filter through the old ones as well.

[mcapra]

A Logstash filter rule could catch these "negative" values and append a field or tag to your message using a mutate step. Some sort of tag like is_negative would be easy enough to search on. Then you're not bound by the constraints of the standard analyzer for this particular use case.

[cdienger]

Using mcapra's(thanks!) suggestion a filter like this could be used to tag messages containing "assignmentTotal:-' "

Code: Select all

if [message] =~ 'assignmentTotal:-'{
mutate {
add_tag => 'is_negative'
}
}

[mliverez]

Thanks @cdienger this is helpful.

[scottwilkerson]

Let us know if we can be of further assistance