Server slow performance and grok issue
Server slow performance and grok issue
Hi,
We are currently evaluating this product for a month now and having some issue looking to resolve.
Recently the log server got very slow response like login takes more than 5 mins to go into dashboard and can't response for searching.
After server restart it returns to normal but issue happened again & again after a period of time, usually likes 1 day later.
And when I apply some input filter. It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?
Thanks.
Regards,
Tela
We are currently evaluating this product for a month now and having some issue looking to resolve.
Recently the log server got very slow response like login takes more than 5 mins to go into dashboard and can't response for searching.
After server restart it returns to normal but issue happened again & again after a period of time, usually likes 1 day later.
And when I apply some input filter. It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?
Thanks.
Regards,
Tela
- tacolover101
- Posts: 432
- Joined: Mon Apr 10, 2017 11:55 am
Re: Server slow performance and grok issue
can you post a profile and/or screenshots of your NLS performance pages?
how much data is currently open in your indices?
i suspect you're overloaded resources somewhere, and something can't keep up with java. oh java.
how much data is currently open in your indices?
i suspect you're overloaded resources somewhere, and something can't keep up with java. oh java.
Re: Server slow performance and grok issue
I second the overloaded resources suggestion. Is this install from the OVA? If so the default memory setting is only 2Gigs which can quickly become a bottleneck and often needs an increase. Beyond that the info requested by @tacolover101 would be good. Feel free to PM me the profile if you don't want to post it here.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Server slow performance and grok issue
Sorry for late reply.
I just capture current page during system is healthy.
Will try to capture one when system is slow next time.
https://imgur.com/a/AKGvP
Log size is variable from around 6GB to 21 GB per day, average is around 10GB.
I just capture current page during system is healthy.
Will try to capture one when system is slow next time.
https://imgur.com/a/AKGvP
Log size is variable from around 6GB to 21 GB per day, average is around 10GB.
Re: Server slow performance and grok issue
Sorry forgot to mention it is come from ova and I adjust the VM to 4 core and 12GB Memory running in SATA disk.
I was changed to SSD one to test but seems same issue occur.
I was changed to SSD one to test but seems same issue occur.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Server slow performance and grok issue
A SSD should help but with 550,000,000+ documents and just one instance with 12GB of RAM the system is really short on resources.
I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.
I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.
Re: Server slow performance and grok issue
What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact:
Re: Server slow performance and grok issue
In order to give a recommendation, I would need to know what you expect the peak messages per daytela wrote:What is the recommended configuration / setup /number of servers for current amount of log?
And do you know the issue with grok is expected behaviour or not?
Thanks for your suggestion.
At a minimum, as I mentioned earlier
scottwilkerson wrote:I would strongly suggest planning a proper cluster with several instances in the cluster both to share the load and provide redundancy, also SSD's will help write and read the volume of data you have.
Re: Server slow performance and grok issue
Nagios Log Server uses Logstash under the hood for it's message parsing. The grok filter Logstash plugin uses Oniguruma for its regex library, which does indeed support lookaheads/behinds as described here:tela wrote:It seems regex positive lookbehind/lookahead (?<=etc) is support in grok debugger but not in nagios log server, is it normal?
Code: Select all
(?=subexp) look-ahead
(?!subexp) negative look-ahead
(?<=subexp) look-behind
(?<!subexp) negative look-behind
Subexp of look-behind must be fixed-width.
But top-level alternatives can be of various lengths.
ex. (?<=a|bc) is OK. (?<=aaa(?:b|cd)) is not allowed.
In negative look-behind, capturing group isn't allowed,
but non-capturing group (?:) is allowed.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
-
- DevOps Engineer
- Posts: 19396
- Joined: Tue Nov 15, 2011 3:11 pm
- Location: Nagios Enterprises
- Contact: