gearman : Too many open files

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Sisa
Posts: 10
Joined: Mon Oct 10, 2016 7:44 am

Re: gearman : Too many open files

Post by Sisa »

Hello,

We set a little script that will catch the open files by gearmand when it will crash. Then, we will find the corrupted mod-gearman.
we heard about the memory leak of a version of mod-gearman and we already changed the version.

Wait and see and thank you again.
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: gearman : Too many open files

Post by rkennedy »

Sounds good - let us know how your testing goes.
Former Nagios Employee
Sisa
Posts: 10
Joined: Mon Oct 10, 2016 7:44 am

Re: gearman : Too many open files

Post by Sisa »

Hello guys,

gearmand has crashed twice and we catched the FD of the process.
I though a worker was the root cause and I hoped that I could find it with lsof but I can't.
Actualy, I got a lot of lines like this one :

Code: Select all

gearmand 31466 gearmand *927u  sock                0,6          0t0 1055092177 can't identify protocol
the number of open files (by gearmand) :

Code: Select all

[root@hpcnagios:~/monitoringGearmand]# wc -l casA
22016 casA
casA is the output of the lsof command while gearmand was crashed

the number of lines with "can't identify protocol :

Code: Select all

[root@hpcnagios:~/monitoringGearmand]# grep -c "identify protocol" casA
19214
Anyone already got this behavior ?

Thank you,
Sisa
avandemore
Posts: 1597
Joined: Tue Sep 27, 2016 4:57 pm

Re: gearman : Too many open files

Post by avandemore »

Please post the results of this:

Code: Select all

# cat /proc/sys/fs/file-max
# su - <your user name>
$ ulimit -Hn
$ ulimit -Sn
Also what are the contents of /etc/security/limits.conf?
Previous Nagios employee
Sisa
Posts: 10
Joined: Mon Oct 10, 2016 7:44 am

Re: gearman : Too many open files

Post by Sisa »

Hello,

/proc/sys/fs/file-max :
6514861

the limit (hard and soft) of the user (gearmand) running gearmand is : 22000
the contents of /etc/security/limits.conf :

Code: Select all

# >= nombre de queues x taille des queues
gearmand		hard	nofile		22000
gearmand		soft	nofile		22000

root		hard	nofile		22000 
root		soft	nofile		22000
# End of file
avandemore
Posts: 1597
Joined: Tue Sep 27, 2016 4:57 pm

Re: gearman : Too many open files

Post by avandemore »

I don't think raising the open file limit is a bad thing. Relatively speaking, 22000 isn't all that many open sockets for a user doing a lot of networking although I'm unfamiliar with your specific setup. If it was me, I'd try raising it to 50000 or more and see what happens.

When the system is in a bad state what is the output of:

Code: Select all

ss -s
Previous Nagios employee
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: gearman : Too many open files

Post by tgriep »

I found this link that talks about too many network connections to the server could cause the same issue. You may want to increase that as well.
https://groups.google.com/forum/#!topic ... nGlRdYQeDM
Be sure to check out our Knowledgebase for helpful articles and solutions!
Sisa
Posts: 10
Joined: Mon Oct 10, 2016 7:44 am

Re: gearman : Too many open files

Post by Sisa »

Hello,

Sorry for the belated reply.
In our case, increasing the max number of FD didn't fix the problem. Gearmand still crashed after increasing to 33000.

Today, gearmands seems to work properly.

Below, settings we have modified :
- we increased the number of gearmand threads with the -t option (up to 8 threads)
- on the "worker" servers, we have modified the follow parameters in order to reduce the number of worker processes :
* reducing min-worker to 10
* reducing max-worker to 50
* increasing idle-timeout to 600 instead of 60 : so we have reduced the number of connection to the job server

Actually, we think the huge number of FD is a consequence and not the root cause

Thank you everybody for your support again.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: gearman : Too many open files

Post by tgriep »

Your welcome. If you don't have any more questions, shall I close and lock the post for you?
Be sure to check out our Knowledgebase for helpful articles and solutions!
Sisa
Posts: 10
Joined: Mon Oct 10, 2016 7:44 am

Re: gearman : Too many open files

Post by Sisa »

I think you can. Thank you again :D
Locked