gearman - a lot JOBs waiting

[rkennedy]

Can you please PM over your profile for us to take a deeper look at?

[bosecorp]

before we head into that direction. I don;t see this problem in my other gearman servers. I don't see this problem on my master gearman server

[bheden]

Can you please post the contents of /etc/mod_gearman/mod_gearman_neb.conf and /var/log/gearmand/gearmand.log?

You might try increasing the value of "result_workers" in the meantime. It looks specifically like its the check_results queue that is hung up, but the hostgroup worker queue is executing properly.

[bosecorp]

Increased # workers, but I still see the same issue

[bheden]

Can you please post the contents of /etc/mod_gearman/mod_gearman_neb.conf and /var/log/gearmand/gearmand.log?

Did you increase result_workers (via result_workers in the neb configuration) or did you increase workers (via min-worker and max-worker in the worker configuration)? These are two different things.

[bosecorp]

via the worker.....the problem is not with the NEB module, it's with only one worker

for some reason, today seems to be fine.......I am going to monitor this worker today and tomorrow

[rkennedy]

That's good news, let us know how your testing goes.

[bosecorp]

spoke too soon.

I see the # jobs waiting going up, but then goes back down.

[bheden]

Still just in reference to the results queue?

Did you ever increase the results worker in the neb configuration? If the result worker can't keep up with the results for whatever reason, you need to increase that value.

Also -

I started noticing that my gearman server is getting behind on JOBs

seems to be having a hard time keeping up

So what is the problem exactly? Are you seeing any checks latency? Are you seeing improper/stale checks come back in Nagios? Or does it just *seem* like its taking longer than it should? Is there a value that the results worker queue sticks around usually, or does it continuously increase in size (and stay increasing in size)?

[bosecorp]

the issues is that some times I see the jobs waiting queue building up