Mod Gearman Performance Issues Nagios 2014R2.6

[cmerchant]

Could you supply us with the rest of the performance metrics? Admin --> Monitoring Engine Status --> and take screen shot of the whole screen

specifically Monitoring Engine Check Statistics and Monitoring Engine Performance.

I would be interested in the service time avg and service time latency.. Thanks.

I'm sure you are referring to this document:

http://nagios.sourceforge.net/docs/nagi ... weaks.html

[rajasegar]

Could you supply us with the rest of the performance metrics? Admin --> Monitoring Engine Status --> and take screen shot of the whole screen

specifically Monitoring Engine Check Statistics and Monitoring Engine Performance.

I would be interested in the service time avg and service time latency.. Thanks.

I'm sure you are referring to this document:

http://nagios.sourceforge.net/docs/nagi ... weaks.html

http://albertolarripa.com/2013/03/03/nagios-tuning/

2015-04-01_07-14-42.png

[Box293]

Please advise if the following will help our performance

use_large_installation_tweaks=1 (already in place)
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
check_result_reaper_frequency=3
max_check_result_reaper_time=10

I've performed load testing before and I've found that these two settings do give better performance:

check_result_reaper_frequency=3
max_check_result_reaper_time=10

"Better performance" is somewhat hard to measure but I did notice the load average drop.

This is outlined in: http://assets.nagios.com/downloads/nagi ... rmance.pdf

[rajasegar]

Please advise if the following will help our performance

use_large_installation_tweaks=1 (already in place)
enable_environment_macros=0
free_child_process_memory=0
child_processes_fork_twice=0
check_result_reaper_frequency=3
max_check_result_reaper_time=10

I've performed load testing before and I've found that these two settings do give better performance:

check_result_reaper_frequency=3
max_check_result_reaper_time=10

"Better performance" is somewhat hard to measure but I did notice the load average drop.

This is outlined in: http://assets.nagios.com/downloads/nagi ... rmance.pdf

Thanks. Will try it out.

[cmerchant]

Thank box293 for that info. OP, Let us know how those parameters work for you. We'll leave this thread open for now.

[rajasegar]

Thank box293 for that info. OP, Let us know how those parameters work for you. We'll leave this thread open for now.

No difference since applying the parameters.

Here is the latest situation after adding 3 load balancers.

2015-04-07_16-21-45.png

2015-04-07_16-20-23.png

What is screwing up the scheduling rate?

For comparison here is another instance with a quarter of the load and no gearman.

2015-04-07_16-21-05.png

[rajasegar]

Now the graph is practically empty. Is this some kind of display issue?
How do I check if everything is ok?

2015-04-07_16-39-21.png

2015-04-07_16-39-33.png

[scottwilkerson]

One change that happened between these versions was a change to the auto_rescheduling_window nagios.cfg setting

The previous setting was

Code: Select all

auto_rescheduling_window=180

I would change this back to 180 as it definitely could be causing the issues if you have some longer running mod_gearman checks

then restart nagios

Code: Select all

service nagios restart

[rajasegar]

One change that happened between these versions was a change to the auto_rescheduling_window nagios.cfg setting

The previous setting was

Code: Select all

auto_rescheduling_window=180

I would change this back to 180 as it definitely could be causing the issues if you have some longer running mod_gearman checks

then restart nagios

Code: Select all

service nagios restart

Yes, we do have some long running checks.
Will try this setting out out now.
Currently it is set as 45.

[rajasegar]

Same problem. The load also shot up considerably.

2015-04-08_07-32-52.png