Page 2 of 2
Re: gearman : Too many open files
Posted: Wed Jan 25, 2017 4:03 am
by Sisa
Hello,
We set a little script that will catch the open files by gearmand when it will crash. Then, we will find the corrupted mod-gearman.
we heard about the memory leak of a version of mod-gearman and we already changed the version.
Wait and see and thank you again.
Re: gearman : Too many open files
Posted: Wed Jan 25, 2017 10:50 am
by rkennedy
Sounds good - let us know how your testing goes.
Re: gearman : Too many open files
Posted: Fri Feb 24, 2017 10:02 am
by Sisa
Hello guys,
gearmand has crashed twice and we catched the FD of the process.
I though a worker was the root cause and I hoped that I could find it with lsof but I can't.
Actualy, I got a lot of lines like this one :
Code: Select all
gearmand 31466 gearmand *927u sock 0,6 0t0 1055092177 can't identify protocol
the number of open files (by gearmand) :
Code: Select all
[root@hpcnagios:~/monitoringGearmand]# wc -l casA
22016 casA
casA is the output of the lsof command while gearmand was crashed
the number of lines with "can't identify protocol :
Code: Select all
[root@hpcnagios:~/monitoringGearmand]# grep -c "identify protocol" casA
19214
Anyone already got this behavior ?
Thank you,
Sisa
Re: gearman : Too many open files
Posted: Fri Feb 24, 2017 3:26 pm
by avandemore
Please post the results of this:
Code: Select all
# cat /proc/sys/fs/file-max
# su - <your user name>
$ ulimit -Hn
$ ulimit -Sn
Also what are the contents of
/etc/security/limits.conf?
Re: gearman : Too many open files
Posted: Tue Feb 28, 2017 3:08 am
by Sisa
Hello,
/proc/sys/fs/file-max :
6514861
the limit (hard and soft) of the user (gearmand) running gearmand is : 22000
the contents of /etc/security/limits.conf :
Code: Select all
# >= nombre de queues x taille des queues
gearmand hard nofile 22000
gearmand soft nofile 22000
root hard nofile 22000
root soft nofile 22000
# End of file
Re: gearman : Too many open files
Posted: Tue Feb 28, 2017 5:35 pm
by avandemore
I don't think raising the open file limit is a bad thing. Relatively speaking, 22000 isn't all that many open sockets for a user doing a lot of networking although I'm unfamiliar with your specific setup. If it was me, I'd try raising it to 50000 or more and see what happens.
When the system is in a bad state what is the output of:
Re: gearman : Too many open files
Posted: Tue Feb 28, 2017 5:53 pm
by tgriep
I found this link that talks about too many network connections to the server could cause the same issue. You may want to increase that as well.
https://groups.google.com/forum/#!topic ... nGlRdYQeDM
Re: gearman : Too many open files
Posted: Tue May 30, 2017 4:29 am
by Sisa
Hello,
Sorry for the belated reply.
In our case, increasing the max number of FD didn't fix the problem. Gearmand still crashed after increasing to 33000.
Today, gearmands seems to work properly.
Below, settings we have modified :
- we increased the number of gearmand threads with the -t option (up to 8 threads)
- on the "worker" servers, we have modified the follow parameters in order to reduce the number of worker processes :
* reducing min-worker to 10
* reducing max-worker to 50
* increasing idle-timeout to 600 instead of 60 : so we have reduced the number of connection to the job server
Actually, we think the huge number of FD is a consequence and not the root cause
Thank you everybody for your support again.
Re: gearman : Too many open files
Posted: Tue May 30, 2017 2:02 pm
by tgriep
Your welcome. If you don't have any more questions, shall I close and lock the post for you?
Re: gearman : Too many open files
Posted: Tue Jun 06, 2017 10:22 am
by Sisa
I think you can. Thank you again
