Page 1 of 2
Server slow performance and grok issue
Posted: Wed Mar 28, 2018 10:42 pm
by tela
Hi,
We are currently evaluating this product for a month now and having some issue looking to resolve.
Recently the log server got very slow response like login takes more than 5 mins to go into dashboard and can't response for searching.
After server restart it returns to normal but issue happened again & again after a period of time, usually likes 1 day later.
And when I apply some input filter. It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?
Thanks.
Regards,
Tela
Re: Server slow performance and grok issue
Posted: Thu Mar 29, 2018 10:32 am
by tacolover101
can you post a profile and/or screenshots of your NLS performance pages?
how much data is currently open in your indices?
i suspect you're overloaded resources somewhere, and something can't keep up with java. oh java.
Re: Server slow performance and grok issue
Posted: Thu Mar 29, 2018 1:31 pm
by cdienger
I second the overloaded resources suggestion. Is this install from the OVA? If so the default memory setting is only 2Gigs which can quickly become a bottleneck and often needs an increase. Beyond that the info requested by
@tacolover101 would be good. Feel free to PM me the profile if you don't want to post it here.
Re: Server slow performance and grok issue
Posted: Tue Apr 10, 2018 9:53 pm
by tela
Sorry for late reply.
I just capture current page during system is healthy.
Will try to capture one when system is slow next time.
https://imgur.com/a/AKGvP
Log size is variable from around 6GB to 21 GB per day, average is around 10GB.
Re: Server slow performance and grok issue
Posted: Tue Apr 10, 2018 9:57 pm
by tela
Sorry forgot to mention it is come from ova and I adjust the VM to 4 core and 12GB Memory running in SATA disk.
I was changed to SSD one to test but seems same issue occur.
Re: Server slow performance and grok issue
Posted: Wed Apr 11, 2018 2:18 pm
by scottwilkerson
A SSD should help but with 550,000,000+ documents and just one instance with 12GB of RAM the system is really short on resources.
I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.
Re: Server slow performance and grok issue
Posted: Wed Apr 11, 2018 9:32 pm
by tela
What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.
Re: Server slow performance and grok issue
Posted: Thu Apr 12, 2018 7:21 am
by scottwilkerson
tela wrote:What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.
In order to give a recommendation, I would need to know what you expect the peak messages per day
At a minimum, as I mentioned earlier
scottwilkerson wrote:I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.
Re: Server slow performance and grok issue
Posted: Thu Apr 12, 2018 8:31 am
by mcapra
tela wrote:It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?
Nagios Log Server uses Logstash under the hood for it's message parsing. The
grok filter Logstash plugin uses
Oniguruma for its regex library, which does indeed support lookaheads/behinds as described here:
Code: Select all
(?=subexp) look-ahead
(?!subexp) negative look-ahead
(?<=subexp) look-behind
(?<!subexp) negative look-behind
Subexp of look-behind must be fixed-width.
But top-level alternatives can be of various lengths.
ex. (?<=a|bc) is OK. (?<=aaa(?:b|cd)) is not allowed.
In negative look-behind, capturing group isn't allowed,
but non-capturing group (?:) is allowed.
We'd need to see the exact grok rule you're applying as well as a sample log message to identify any sort of mis-match between the third party
grok debugger and what actually happens within the
grok filter plugin.
Re: Server slow performance and grok issue
Posted: Thu Apr 12, 2018 2:26 pm
by scottwilkerson