Page 2 of 3

Re: gearman - a lot JOBs waiting

Posted: Fri Mar 04, 2016 3:23 pm
by rkennedy
Can you please PM over your profile for us to take a deeper look at?

Re: gearman - a lot JOBs waiting

Posted: Fri Mar 04, 2016 7:05 pm
by bosecorp
before we head into that direction. I don;t see this problem in my other gearman servers. I don't see this problem on my master gearman server

Re: gearman - a lot JOBs waiting

Posted: Mon Mar 07, 2016 2:20 pm
by bheden
Can you please post the contents of /etc/mod_gearman/mod_gearman_neb.conf and /var/log/gearmand/gearmand.log?

You might try increasing the value of "result_workers" in the meantime. It looks specifically like its the check_results queue that is hung up, but the hostgroup worker queue is executing properly.

Re: gearman - a lot JOBs waiting

Posted: Sun Apr 03, 2016 8:53 pm
by bosecorp
Increased # workers, but I still see the same issue

Re: gearman - a lot JOBs waiting

Posted: Mon Apr 04, 2016 9:18 am
by bheden
Can you please post the contents of /etc/mod_gearman/mod_gearman_neb.conf and /var/log/gearmand/gearmand.log?
Did you increase result_workers (via result_workers in the neb configuration) or did you increase workers (via min-worker and max-worker in the worker configuration)? These are two different things.

Re: gearman - a lot JOBs waiting

Posted: Tue Apr 05, 2016 9:56 am
by bosecorp
via the worker.....the problem is not with the NEB module, it's with only one worker

for some reason, today seems to be fine.......I am going to monitor this worker today and tomorrow

Re: gearman - a lot JOBs waiting

Posted: Tue Apr 05, 2016 11:33 am
by rkennedy
That's good news, let us know how your testing goes.

Re: gearman - a lot JOBs waiting

Posted: Tue Apr 05, 2016 5:39 pm
by bosecorp
spoke too soon.

I see the # jobs waiting going up, but then goes back down.

Re: gearman - a lot JOBs waiting

Posted: Wed Apr 06, 2016 9:54 am
by bheden
Still just in reference to the results queue?

Did you ever increase the results worker in the neb configuration? If the result worker can't keep up with the results for whatever reason, you need to increase that value.

Also -
I started noticing that my gearman server is getting behind on JOBs

seems to be having a hard time keeping up
So what is the problem exactly? Are you seeing any checks latency? Are you seeing improper/stale checks come back in Nagios? Or does it just *seem* like its taking longer than it should? Is there a value that the results worker queue sticks around usually, or does it continuously increase in size (and stay increasing in size)?

Re: gearman - a lot JOBs waiting

Posted: Wed Apr 06, 2016 11:54 am
by bosecorp
the issues is that some times I see the jobs waiting queue building up