Page 2 of 3

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Mar 31, 2015 5:10 pm
by cmerchant
Could you supply us with the rest of the performance metrics? Admin --> Monitoring Engine Status --> and take screen shot of the whole screen

specifically Monitoring Engine Check Statistics and Monitoring Engine Performance.

I would be interested in the service time avg and service time latency.. Thanks.

I'm sure you are referring to this document:

http://nagios.sourceforge.net/docs/nagi ... weaks.html

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Mar 31, 2015 6:15 pm
by rajasegar
cmerchant wrote:Could you supply us with the rest of the performance metrics? Admin --> Monitoring Engine Status --> and take screen shot of the whole screen

specifically Monitoring Engine Check Statistics and Monitoring Engine Performance.

I would be interested in the service time avg and service time latency.. Thanks.

I'm sure you are referring to this document:

http://nagios.sourceforge.net/docs/nagi ... weaks.html
http://albertolarripa.com/2013/03/03/nagios-tuning/
2015-04-01_07-14-42.png

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Mar 31, 2015 10:03 pm
by Box293
rajasegar wrote:Please advise if the following will help our performance

use_large_installation_tweaks=1 (already in place)
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
check_result_reaper_frequency=3
max_check_result_reaper_time=10
I've performed load testing before and I've found that these two settings do give better performance:

check_result_reaper_frequency=3
max_check_result_reaper_time=10

"Better performance" is somewhat hard to measure but I did notice the load average drop.

This is outlined in: http://assets.nagios.com/downloads/nagi ... rmance.pdf

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Mar 31, 2015 10:29 pm
by rajasegar
Box293 wrote:
rajasegar wrote:Please advise if the following will help our performance

use_large_installation_tweaks=1 (already in place)
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
check_result_reaper_frequency=3
max_check_result_reaper_time=10
I've performed load testing before and I've found that these two settings do give better performance:

check_result_reaper_frequency=3
max_check_result_reaper_time=10

"Better performance" is somewhat hard to measure but I did notice the load average drop.

This is outlined in: http://assets.nagios.com/downloads/nagi ... rmance.pdf
Thanks. Will try it out.

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Wed Apr 01, 2015 9:47 am
by cmerchant
Thank box293 for that info. OP, Let us know how those parameters work for you. We'll leave this thread open for now.

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Apr 07, 2015 3:25 am
by rajasegar
cmerchant wrote:Thank box293 for that info. OP, Let us know how those parameters work for you. We'll leave this thread open for now.
No difference since applying the parameters.

Here is the latest situation after adding 3 load balancers.
2015-04-07_16-21-45.png
2015-04-07_16-20-23.png
What is screwing up the scheduling rate?

For comparison here is another instance with a quarter of the load and no gearman.
2015-04-07_16-21-05.png

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Apr 07, 2015 3:43 am
by rajasegar
Now the graph is practically empty. Is this some kind of display issue?
How do I check if everything is ok?
2015-04-07_16-39-21.png
2015-04-07_16-39-33.png

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Apr 07, 2015 1:45 pm
by scottwilkerson
One change that happened between these versions was a change to the auto_rescheduling_window nagios.cfg setting

The previous setting was

Code: Select all

auto_rescheduling_window=180
I would change this back to 180 as it definitely could be causing the issues if you have some longer running mod_gearman checks

then restart nagios

Code: Select all

service nagios restart

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Apr 07, 2015 6:10 pm
by rajasegar
scottwilkerson wrote:One change that happened between these versions was a change to the auto_rescheduling_window nagios.cfg setting

The previous setting was

Code: Select all

auto_rescheduling_window=180
I would change this back to 180 as it definitely could be causing the issues if you have some longer running mod_gearman checks

then restart nagios

Code: Select all

service nagios restart

Yes, we do have some long running checks.
Will try this setting out out now.
Currently it is set as 45.

Re: Mod Gearman Performance Issues Nagios 2014R2.6

Posted: Tue Apr 07, 2015 6:34 pm
by rajasegar
Same problem. The load also shot up considerably.
2015-04-08_07-32-52.png