Filter Questions and exact matching?

[Jklre]

So I have a few questions. I'm setting up a filer for some new services. We have a json input where we are filtering by a specific field. We have noticed filters do not do exact matches and we are running into some difficulties. We have some services with similar names and its incorrectly matching.

For example we have a service named:
ENTERPRISE-DOCSTORENAS-SERVICE

but it will also match another service
ENTERPRISE-DOCSTORENAS-SERVICE-TEST

When trying to filter out the ENTERPRISE-DOCSTORENAS-SERVICE messages from the ENTERPRISE-DOCSTORENAS-SERVICE-TEST messages filters both.

filter1.jpg

filter2.jpg

Any advice on setting these filters?

[eloyd]

Try changing to a regular expression search and make sure you anchor text correctly. You can find some tips at https://www.elastic.co/guide/en/elastic ... query.html

[hsmith]

Thanks Eric. @Jklre, does that clear it up?

[Jklre]

Try changing to a regular expression search and make sure you anchor text correctly. You can find some tips at https://www.elastic.co/guide/en/elastic ... query.html

Can you use regex in filters? or just in queries. So far we are just filtering by a field with must or mustnot. I can always do a regex querry for the exact term i want but I read somewhere that using filters is much less resource intensive than doing queries.

[eloyd]

Oooooh. You know what, I misread. No, you cannot do regexp in filters, only queries. But I would still use a query for what you want. Here's why:

Filters filter the data so you only see a sub-set of it. Filtering needs to be fast, since every further operation done with the data must pass the filter first.

Once you have a subset of data, queries search for things you are interested in. Optionally, they are used to display different data in different colors on the dashboards. All data that the queries search must first have passed through the filter.

So you could filter on ENTERPRISE-DOCSTORENAS-SERVICE, like you are today. Then Query for you regexp "^ENTERPRISE-DOCSTORENAS-SERVICE$" (or whatever is appropriate for what you're looking for. At least this way, you're only querying a subset of the overall data.


Advanced topic:
You could use the "filter" in Global Configuration to do a grok filter expansion to split your text into multiple fields using pattern matching. Then ENTERPRISE-DOCSTORENAS-SERVICE would be separate from ENTERPRISE-DOCSTORENAS-SERVICE-TEST because your pattern match would know to look for spaces to separate the field. Then you could filter on the field matching ENTERPRISE-DOCSTORENAS-SERVICE and not worry about the query.

[jolson]

While using queries in Elasticsearch, the default nature is to match the string provided ("ENTERPRISE-DOCSTORENAS-SERVICE" in your case) and expand infinitely outward from there. The problem arises because your log messages are so similar, and because ENTERPRISE-DOCSTORENAS-SERVICE-NAS is longer than ENTERPRISE-DOCSTORENAS-SERVICE.

Given the below information, how do we best resolve this?

-ENTERPRISE-DOCSTORENAS-SERVICE-NAS is what we want to see
-ENTERPRISE-DOCSTORENAS-SERVICE is what we'd like to filter out
-We'd like to use filters so that the dashboard is more logical.

This actually worked on my system without issue:

2016-03-24 16_50_04-Dashboard • Nagios Log Server - Firefox Developer Edition.png

2016-03-24 16_50_38-Dashboard • Nagios Log Server - Firefox Developer Edition.png

Can you point out where I might have gone wrong/done things differently with regards to my testing?

[eloyd]

Now you have to filter out everything you don't want. Ugly and not very scalable. Unless you know that everything you don't want matches a predictable pattern and can filter out based on that.

[hsmith]

Eric, do you have a suggestion to make it 'better'?

[eloyd]

I would use a grok filter to match fields. Harder to set up but easier to maintain in the future with more flexibility.

[hsmith]

I agree with this, assuming the logs are all the same format.