Server slow performance and grok issue

[tela]

Hi,

We are currently evaluating this product for a month now and having some issue looking to resolve.
Recently the log server got very slow response like login takes more than 5 mins to go into dashboard and can't response for searching.
After server restart it returns to normal but issue happened again & again after a period of time, usually likes 1 day later.

And when I apply some input filter. It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?

Thanks.

Regards,
Tela

[tacolover101]

can you post a profile and/or screenshots of your NLS performance pages?

how much data is currently open in your indices?

i suspect you're overloaded resources somewhere, and something can't keep up with java. oh java.

[cdienger]

I second the overloaded resources suggestion. Is this install from the OVA? If so the default memory setting is only 2Gigs which can quickly become a bottleneck and often needs an increase. Beyond that the info requested by @tacolover101 would be good. Feel free to PM me the profile if you don't want to post it here.

[tela]

Sorry for late reply.
I just capture current page during system is healthy.
Will try to capture one when system is slow next time.
https://imgur.com/a/AKGvP

Log size is variable from around 6GB to 21 GB per day, average is around 10GB.

[tela]

Sorry forgot to mention it is come from ova and I adjust the VM to 4 core and 12GB Memory running in SATA disk.
I was changed to SSD one to test but seems same issue occur.

[scottwilkerson]

A SSD should help but with 550,000,000+ documents and just one instance with 12GB of RAM the system is really short on resources.

I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.

[tela]

What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.

[scottwilkerson]

What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.

In order to give a recommendation, I would need to know what you expect the peak messages per day

At a minimum, as I mentioned earlier

I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.

[mcapra]

It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?

Nagios Log Server uses Logstash under the hood for it's message parsing. The grok filter Logstash plugin uses Oniguruma for its regex library, which does indeed support lookaheads/behinds as described here:

Code: Select all

  (?=subexp)         look-ahead
  (?!subexp)         negative look-ahead
  (?<=subexp)        look-behind
  (?<!subexp)        negative look-behind

                     Subexp of look-behind must be fixed-width.
                     But top-level alternatives can be of various lengths.
                     ex. (?<=a|bc) is OK. (?<=aaa(?:b|cd)) is not allowed.

                     In negative look-behind, capturing group isn't allowed,
                     but non-capturing group (?:) is allowed.

We'd need to see the exact grok rule you're applying as well as a sample log message to identify any sort of mis-match between the third party grok debugger and what actually happens within the grok filter plugin.

[scottwilkerson]

Thanks @mcapra