Hi,
I used mod-gearman for distribute monitoring.
I set max_worker=2048 but when I run gearman_top2 I found that the maximum number of worker available limited to 1024. as I monitored the resource of my gearman server and workers, there are no limitation from hardware resources. Also, as I checked, it can be due to limitation of linux kernel for open files. is it possible to guide me how can I increase this limitation?
for additional info:
I have Nagios XI 5.4.13
mod-gearman 2.1.1-1 with server version 0.33
Centos 6 for gearman server
4 Workers on Centos 7
number of hosts about 12K
thanks
Mod-gearman max workers limit to 1024
Re: Mod-gearman max workers limit to 1024
Take a look at this link that has instructions for increasing the open file limits on a Linux system.
https://easyengine.io/tutorials/linux/i ... les-limit/
If you have any questions, let us know.
https://easyengine.io/tutorials/linux/i ... les-limit/
If you have any questions, let us know.
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: Mod-gearman max workers limit to 1024
You can use "ulimit -a" to check limits on the system, but as https://github.com/sni/mod_gearman/issues/65 and the author of mod_gearman point out, increasing the max-workers beyond 1024 can create problems and isn't typically needed.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Re: Mod-gearman max workers limit to 1024
I had 4 workers with below hardware spec on each one:
10 core CPU
10 GB RAM
but worker servers can't use all mentioned hardware because of this limitation. I decreased the amount of Hardware on each workers and create new workers to recover this issue and now I have 8 worker servers.
I wrote a plugin to check host availability and because of numerous ping packets for each host (60 packets in each 1 minute) the worker threads will not be freed in timely manner. In such cases we need more threads to handle the tasks in the queues.
Seems, this case need to be attend by author of mod-gearman.
thanks
10 core CPU
10 GB RAM
but worker servers can't use all mentioned hardware because of this limitation. I decreased the amount of Hardware on each workers and create new workers to recover this issue and now I have 8 worker servers.
I wrote a plugin to check host availability and because of numerous ping packets for each host (60 packets in each 1 minute) the worker threads will not be freed in timely manner. In such cases we need more threads to handle the tasks in the queues.
Seems, this case need to be attend by author of mod-gearman.
thanks
Re: Mod-gearman max workers limit to 1024
At that scale, I think containerizing your worker servers, deploying more of them, and scaling down the worker threads per worker server would solve an awful lot of problems. Especially based on the lack of activity in the GitHub issue @cdienger mentioned.
These days, it's often easier and cheaper to work around un-optimized code rather than spending resources optimizing when you have the option. Though sometimes you simply don't have that option
These days, it's often easier and cheaper to work around un-optimized code rather than spending resources optimizing when you have the option. Though sometimes you simply don't have that option
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Mod-gearman max workers limit to 1024
@mcapra you are right. but are you sure mod-gearman is compatible with containerization technologies such as Docker?
Also, we are distributing service using mod-gearman and using containers can cause increasing complexity in case of troubleshooting issues.
Also, we are distributing service using mod-gearman and using containers can cause increasing complexity in case of troubleshooting issues.
Re: Mod-gearman max workers limit to 1024
I don't see why it wouldn't be compatible. You'd just need to be mindful of any dependencies your Nagios plugins may require.salami wrote:are you sure mod-gearman is compatible with containerization technologies such as Docker?
Here's a POC implementation of stuffing a gearman worker into a container:
https://github.com/bernardoVale/docker- ... man-worker
I imagine stuffing a gearman job server inside a container would look very similar, though with more networking requirements. Most of the "network requirements" could be handled with a few lines in a Kubernetes deployment.
Having good container orchestration is Step 0 of such a setup. Not every org has those resources which would definitely increase complexity. It would probably be a non-starter in that case.salami wrote:Also, we are distributing service using mod-gearman and using containers can cause increasing complexity in case of troubleshooting issues.
Your orchestration wouldn't need to be some great big beautiful Kubernetes cluster though. Could be as simple as an Ansible playbook or Chef cookbook.
Former Nagios employee
https://www.mcapra.com/
https://www.mcapra.com/
Re: Mod-gearman max workers limit to 1024
Thanks for the input, @mcapra!
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.