Variable limit - LS query

[rferebee]

Good morning Nagios team, I'm wondering if there is a variable limit on queries within Log Server? If you take a look at the attached screenshot, we're currently monitoring for domains flagged as unsafe by MS-ISAC. Unfortunately, the number of domains grows almost weekly and I'm not sure how many I'll be able to add in a single query or if perhaps there's a better way to go about it in your opinion?

Thank you for your help.

[mcapra]

I can't think of a way to solve this that doesn't involve reducing the cardinality of the query. I think the only limiting factor within Elasticsearch is the Netty server's max content length (http.max_content_length) which is like 100mb by default -- it'd have to be a massive Lucene query to hit that limit:
https://www.elastic.co/guide/en/elastic ... -http.html

You could certainly reduce the cardinality with clever filter rules (or input rules), like adding a tag to the messages of those flagged hosts. A giant manually maintained filter rule that needs to hit every single message sounds kinda painful though. Ideally this tagging would be done at the agent level -- you'd update the flagged machines nxlog configs to include that tag before it passes the message to NLS, then query on that special tag instead of an ever growing list of hosts.

[rferebee]

Sorry, I guess the reply I got isn't clear enough for me.

Can I keep adding domains to the query shown in the screenshot or do I need to figure out a better way to accomplish what we need?

[benjaminsmith]

Hi,

It would be best to tag messages based on the domain of interest and then search for the tags. Can you provide a screenshot of the details that this query is currently returning?

A couple of other options, you can limit the number of queries by using domain1|domain2|domain3|domainX in a single query field.

It's also possible to use regex instead of Lucene,

https://www.elastic.co/guide/en/elastic ... exp-syntax

Regards,
Benjamin