we saw in the gearman_top that only a few jobs were started, but a "lot" stayed in the check_results queue
as can be seen below.
We also saw that just a few jobs were started
The primary nagios process was using 100% CPU
Our test environment is simular to production:
~3900 hosts
~52600 services
In the nagios.log we saw events very slowly getting processed.
Code: Select all
$ gearman_top -b
2020-07-21 11:18:05 - localhost:4730 - v1.1.18
Queue Name | Worker Available | Jobs Waiting | Jobs Running
----------------------------------------------------------------------------------------
check_results | 1 | 1466 | 1
eventhandler | 212 | 0 | 0
host | 167 | 0 | 0
hostgroup_hg_worker_bran | 5 | 0 | 0
hostgroup_hg_worker_dock-ka | 10 | 0 | 0
hostgroup_hg_worker_dock-pa | 0 | 0 | 0
hostgroup_hg_worker_dro | 15 | 0 | 0
hostgroup_hg_worker_dsb | 15 | 0 | 0
hostgroup_hg_worker_finq | 15 | 0 | 0
hostgroup_hg_worker_flmc | 15 | 0 | 0
hostgroup_hg_worker_ggn | 2 | 0 | 0
hostgroup_hg_worker_jzhz | 15 | 0 | 0
hostgroup_hg_worker_lek | 15 | 0 | 0
hostgroup_hg_worker_sdb | 15 | 0 | 0
hostgroup_hg_worker_sdzb | 30 | 0 | 0
hostgroup_hg_worker_snms | 115 | 0 | 0
hostgroup_hg_worker_sviz | 0 | 0 | 0
hostgroup_hg_worker_wnf | 10 | 0 | 0
notification | 30 | 0 | 0
service | 167 | 0 | 0
worker_JZHZDCBTCSS-001 | 1 | 0 | 0
worker_ZORG-WAATNMW01 | 1 | 0 | 0
worker_ZORG-WAATNMW02 | 1 | 0 | 0
worker_bmrdhgtcssm02.brandmr.local | 1 | 0 | 0
worker_dsbwaa01pmgw02 | 1 | 0 | 0
worker_flmc-gropnag01.flamco.local | 1 | 0 | 0
worker_gdr01dcbmgw02 | 1 | 0 | 0
worker_monxisltn-vms.dockaas.nl | 1 | 0 | 0
worker_sbhptsssm013.sltn-beheer.local | 1 | 0 | 0
worker_sbhptsssm014.sltn-beheer.local | 1 | 0 | 0
worker_sbhptsssm015.sltn-beheer.local | 1 | 0 | 0
worker_sdb-waatcssm01.dbij.local | 1 | 0 | 0
worker_sr_monsltn_pa.dockaas.nl | 1 | 0 | 0
worker_svizdcbpnagi02.vivium.local | 1 | 0 | 0
worker_svr-lnxngs-002 | 1 | 0 | 0
worker_svr-mgw200 | 1 | 0 | 0
worker_wnf-s-mgw01.wnf.local | 1 | 0 | 0
----------------------------------------------------------------------------------------For a test I unconfigured the mod_geaman broker in the nagios.cfg and restarted nagios.
we then saw in the log events processed much faster, however the status still stayed at Pending.
It looks like problems with gearman are not fixed yet?