gearman : Too many open files
Re: gearman : Too many open files
Hello,
We set a little script that will catch the open files by gearmand when it will crash. Then, we will find the corrupted mod-gearman.
we heard about the memory leak of a version of mod-gearman and we already changed the version.
Wait and see and thank you again.
We set a little script that will catch the open files by gearmand when it will crash. Then, we will find the corrupted mod-gearman.
we heard about the memory leak of a version of mod-gearman and we already changed the version.
Wait and see and thank you again.
Re: gearman : Too many open files
Sounds good - let us know how your testing goes.
Former Nagios Employee
Re: gearman : Too many open files
Hello guys,
gearmand has crashed twice and we catched the FD of the process.
I though a worker was the root cause and I hoped that I could find it with lsof but I can't.
Actualy, I got a lot of lines like this one :
the number of open files (by gearmand) :
casA is the output of the lsof command while gearmand was crashed
the number of lines with "can't identify protocol :
Anyone already got this behavior ?
Thank you,
Sisa
gearmand has crashed twice and we catched the FD of the process.
I though a worker was the root cause and I hoped that I could find it with lsof but I can't.
Actualy, I got a lot of lines like this one :
Code: Select all
gearmand 31466 gearmand *927u sock 0,6 0t0 1055092177 can't identify protocol
Code: Select all
[root@hpcnagios:~/monitoringGearmand]# wc -l casA
22016 casA
the number of lines with "can't identify protocol :
Code: Select all
[root@hpcnagios:~/monitoringGearmand]# grep -c "identify protocol" casA
19214
Thank you,
Sisa
-
- Posts: 1597
- Joined: Tue Sep 27, 2016 4:57 pm
Re: gearman : Too many open files
Please post the results of this:
Also what are the contents of /etc/security/limits.conf?
Code: Select all
# cat /proc/sys/fs/file-max
# su - <your user name>
$ ulimit -Hn
$ ulimit -Sn
Previous Nagios employee
Re: gearman : Too many open files
Hello,
/proc/sys/fs/file-max :
6514861
the limit (hard and soft) of the user (gearmand) running gearmand is : 22000
the contents of /etc/security/limits.conf :
/proc/sys/fs/file-max :
6514861
the limit (hard and soft) of the user (gearmand) running gearmand is : 22000
the contents of /etc/security/limits.conf :
Code: Select all
# >= nombre de queues x taille des queues
gearmand hard nofile 22000
gearmand soft nofile 22000
root hard nofile 22000
root soft nofile 22000
# End of file
-
- Posts: 1597
- Joined: Tue Sep 27, 2016 4:57 pm
Re: gearman : Too many open files
I don't think raising the open file limit is a bad thing. Relatively speaking, 22000 isn't all that many open sockets for a user doing a lot of networking although I'm unfamiliar with your specific setup. If it was me, I'd try raising it to 50000 or more and see what happens.
When the system is in a bad state what is the output of:
When the system is in a bad state what is the output of:
Code: Select all
ss -s
Previous Nagios employee
Re: gearman : Too many open files
I found this link that talks about too many network connections to the server could cause the same issue. You may want to increase that as well.
https://groups.google.com/forum/#!topic ... nGlRdYQeDM
https://groups.google.com/forum/#!topic ... nGlRdYQeDM
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: gearman : Too many open files
Hello,
Sorry for the belated reply.
In our case, increasing the max number of FD didn't fix the problem. Gearmand still crashed after increasing to 33000.
Today, gearmands seems to work properly.
Below, settings we have modified :
- we increased the number of gearmand threads with the -t option (up to 8 threads)
- on the "worker" servers, we have modified the follow parameters in order to reduce the number of worker processes :
* reducing min-worker to 10
* reducing max-worker to 50
* increasing idle-timeout to 600 instead of 60 : so we have reduced the number of connection to the job server
Actually, we think the huge number of FD is a consequence and not the root cause
Thank you everybody for your support again.
Sorry for the belated reply.
In our case, increasing the max number of FD didn't fix the problem. Gearmand still crashed after increasing to 33000.
Today, gearmands seems to work properly.
Below, settings we have modified :
- we increased the number of gearmand threads with the -t option (up to 8 threads)
- on the "worker" servers, we have modified the follow parameters in order to reduce the number of worker processes :
* reducing min-worker to 10
* reducing max-worker to 50
* increasing idle-timeout to 600 instead of 60 : so we have reduced the number of connection to the job server
Actually, we think the huge number of FD is a consequence and not the root cause
Thank you everybody for your support again.
Re: gearman : Too many open files
Your welcome. If you don't have any more questions, shall I close and lock the post for you?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Re: gearman : Too many open files
I think you can. Thank you again