Nagios Support Forum

Hi,

I used mod-gearman for distribute monitoring.
I set max_worker=2048 but when I run gearman_top2 I found that the maximum number of worker available limited to 1024. as I monitored the resource of my gearman server and workers, there are no limitation from hardware resources. Also, as I checked, it can be due to limitation of linux kernel for open files. is it possible to guide me how can I increase this limitation?

for additional info:
I have Nagios XI 5.4.13
mod-gearman 2.1.1-1 with server version 0.33
Centos 6 for gearman server
4 Workers on Centos 7
number of hosts about 12K

thanks

Take a look at this link that has instructions for increasing the open file limits on a Linux system.
https://easyengine.io/tutorials/linux/i ... les-limit/
If you have any questions, let us know.

You can use "ulimit -a" to check limits on the system, but as https://github.com/sni/mod_gearman/issues/65 and the author of mod_gearman point out, increasing the max-workers beyond 1024 can create problems and isn't typically needed.

I had 4 workers with below hardware spec on each one:
10 core CPU
10 GB RAM

but worker servers can't use all mentioned hardware because of this limitation. I decreased the amount of Hardware on each workers and create new workers to recover this issue and now I have 8 worker servers.
I wrote a plugin to check host availability and because of numerous ping packets for each host (60 packets in each 1 minute) the worker threads will not be freed in timely manner. In such cases we need more threads to handle the tasks in the queues.
Seems, this case need to be attend by author of mod-gearman.


thanks

At that scale, I think containerizing your worker servers, deploying more of them, and scaling down the worker threads per worker server would solve an awful lot of problems. Especially based on the lack of activity in the GitHub issue @cdienger mentioned.

These days, it's often easier and cheaper to work around un-optimized code rather than spending resources optimizing when you have the option. Though sometimes you simply don't have that option

@mcapra you are right. but are you sure mod-gearman is compatible with containerization technologies such as Docker?
Also, we are distributing service using mod-gearman and using containers can cause increasing complexity in case of troubleshooting issues.

salami wrote:are you sure mod-gearman is compatible with containerization technologies such as Docker?

I don't see why it wouldn't be compatible. You'd just need to be mindful of any dependencies your Nagios plugins may require.

Here's a POC implementation of stuffing a gearman worker into a container:
https://github.com/bernardoVale/docker- ... man-worker

I imagine stuffing a gearman job server inside a container would look very similar, though with more networking requirements. Most of the "network requirements" could be handled with a few lines in a Kubernetes deployment.

salami wrote:Also, we are distributing service using mod-gearman and using containers can cause increasing complexity in case of troubleshooting issues.

Having good container orchestration is Step 0 of such a setup. Not every org has those resources which would definitely increase complexity. It would probably be a non-starter in that case.

Your orchestration wouldn't need to be some great big beautiful Kubernetes cluster though. Could be as simple as an Ansible playbook or Chef cookbook.

Thanks for the input, @mcapra!