gearman troubles
Posted: Thu Jan 05, 2017 4:48 am
Hello,
We just set up a new CentOS 7 Gearman worker. We seem to experience a weird issue on hosts and services for this Gearman worker node where the next check time is in the past. Is there any issue known to be associated with this? Date and hwclock are correct on both the Nagios server and the gearman workers.
On the Nagios XI server:
On the new Gearman server:
We sometimes also seem to get the error "(host check orphaned, is the mod-gearman worker on queue 'hostgroup_hg_gearman_"
While the service seems to be running on this Gearman worker:
The real time the moment the screenshot was taken was actually 10:48. I have another Gearman 2 worker on CentOS 6 where everything seems to be working fine. The config files are identical, except for the hostgroup whcih defines the checks which have to run on the worker.
Thanks for any help solving this.
We just set up a new CentOS 7 Gearman worker. We seem to experience a weird issue on hosts and services for this Gearman worker node where the next check time is in the past. Is there any issue known to be associated with this? Date and hwclock are correct on both the Nagios server and the gearman workers.
On the Nagios XI server:
Code: Select all
yum list installed | grep gearman
gearmand.x86_64 1:0.33-2 @/gearmand-0.33-2.rhel6.x86_64
gearmand-devel.x86_64 1:0.33-2 @/gearmand-devel-0.33-2.rhel6.x86_64
gearmand-server.x86_64 1:0.33-2 @/gearmand-server-0.33-2.rhel6.x86_64
mod_gearman2.x86_64 2.1.1-1.el6 @/mod_gearman2-2.1.1-1.rhel6.x86_64
Code: Select all
yum list installed | grep gearman [17-01-05 10:40:58]
gearmand.x86_64 1:0.33-2 installed
gearmand-debuginfo.x86_64 1:0.33-2 installed
gearmand-devel.x86_64 1:0.33-2 installed
mod_gearman2.x86_64 2.1.1-1.el7.centos installed
While the service seems to be running on this Gearman worker:
Code: Select all
systemctl status mod-gearman2-worker [17-01-05 10:43:24]
● mod-gearman2-worker.service - Mod-Gearman Worker
Loaded: loaded (/usr/lib/systemd/system/mod-gearman2-worker.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2017-01-05 09:13:31 CET; 1h 30min ago
Docs: http://mod-gearman.org/docs.html
Process: 1162 ExecStart=/usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid (code=exited, status=0/SUCCESS)
Main PID: 1171 (mod_gearman2_wo)
CGroup: /system.slice/mod-gearman2-worker.service
├─1171 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
├─6828 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
├─6830 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
├─6831 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
├─6832 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
├─6833 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
└─6834 /usr/bin/mod_gearman2_worker -d --config=/etc/mod_gearman2/worker.conf --pidfile=/var/mod_gearman2/mod_gearman_worker.pid
Code: Select all
###############################################################################
#
# Mod-Gearman - distribute checks with gearman
#
# Copyright (c) 2010 Sven Nierlein
#
# Worker Module Config
#
###############################################################################
# Identifier, hostname will be used if undefined
#identifier=hostname
# use debug to increase the verbosity of the module.
# Possible values are:
# 0 = only errors
# 1 = debug messages
# 2 = trace messages
# 3 = trace and all gearman related logs are going to stdout.
# Default is 0.
debug=1
# Path to the logfile.
logfile=/var/log/mod_gearman2/mod_gearman_worker.log
# sets the addess of your gearman job server. Can be specified
# more than once to add more server.
server=10.10.10.10:4730
# sets the address of your 2nd (duplicate) gearman job server. Can
# be specified more than once o add more servers.
#dupserver=<host>:<port>
# defines if the worker should execute eventhandlers.
eventhandler=no
# defines if the worker should execute
# service checks.
services=no
# defines if the worker should execute
# host checks.
hosts=no
# sets a list of hostgroups which this worker will work
# on. Either specify a comma seperated list or use
# multiple lines.
#hostgroups=name1
#hostgroups=name2,name3
hostgroups=hg_gearman_2
# sets a list of servicegroups which this worker will
# work on.
#servicegroups=name1,name2,name3
# enables or disables encryption. It is strongly
# advised to not disable encryption. Anybody will be
# able to inject packages to your worker.
# Encryption is enabled by default and you have to
# explicitly disable it.
# When using encryption, you will either have to
# specify a shared password with key=... or a
# keyfile with keyfile=...
# Default is On.
encryption=yes
# A shared password which will be used for
# encryption of data pakets. Should be at least 8
# bytes long. Maximum length is 32 characters.
key=key
# The shared password will be read from this file.
# Use either key or keyfile. Only the first 32
# characters will be used.
#keyfile=/path/to/secret.file
# Path to the pidfile. Usually set by the init script
#pidfile=/var/mod_gearman2/mod_gearman_worker.pid
# Default job timeout in seconds. Currently this value is only used for
# eventhandler. The worker will use the values from the core for host and
# service checks.
job_timeout=60
# Minimum number of worker processes which should
# run at any time.
min-worker=5
# Maximum number of worker processes which should
# run at any time. You may set this equal to
# min-worker setting to disable dynamic starting of
# workers. When setting this to 1, all services from
# this worker will be executed one after another.
max-worker=50
# Time after which an idling worker exists
# This parameter controls how fast your waiting workers will
# exit if there are no jobs waiting.
idle-timeout=30
# Controls the amount of jobs a worker will do before he exits
# Use this to control how fast the amount of workers will go down
# after high load times
max-jobs=1000
# max-age is the threshold for discarding too old jobs. When a new job is older
# than this amount of seconds it will not be executed and just discarded. Set to
# zero to disable this check.
#max-age=0
# defines the rate of spawned worker per second as long
# as there are jobs waiting
spawn-rate=1
# Use this option to disable an extra fork for each plugin execution. Disabling
# this option will reduce the load on the worker host but can lead to problems with
# unclean plugin. Default: yes
fork_on_exec=no
# Set a limit based on the 1min load average. When exceding the load limit,
# no new worker will be started until the current load is below the limit.
# No limit will be used when set to 0.
load_limit1=0
# Same as load_limit1 but for the 5min load average.
load_limit5=0
# Same as load_limit1 but for the 15min load average.
load_limit15=0
# Use this option to show stderr output of plugins too.
# Default: yes
show_error_output=yes
# Use dup_results_are_passive to set if the duplicate result send to the dupserver
# will be passive or active.
# Default is yes (passive).
#dup_results_are_passive=yes
# When embedded perl has been compiled in, you can use this
# switch to enable or disable the embedded perl interpreter.
enable_embedded_perl=on
# Default value used when the perl script does not have a
# "nagios: +epn" or "nagios: -epn" set.
# Perl scripts not written for epn support usually fail with epn,
# so its better to set the default to off.
use_embedded_perl_implicitly=off
# Cache compiled perl scripts. This makes the worker process a little
# bit bigger but makes execution of perl scripts even faster.
# When turned off, Mod-Gearman will still use the embedded perl
# interpreter, but will not cache the compiled script.
use_perl_cache=on
# path to p1 file which is used to execute and cache the
# perl scripts run by the embedded perl interpreter
p1_file=/usr/share/mod_gearman2/mod_gearman_p1.pl
# Security
# restrict_path allows you to restrict this worker to only execute plugins
# from these particular folders. Can be used multiple times to specify more
# than one folder.
# Note that when this restriction is active, no shell will be spawned and
# no shell characters ($`'"()|) are allowed in the command line itself.
#restrict_path=/usr/local/plugins/
# Workarounds
# workaround for rc 25 bug
# duplicate jobs from gearmand result in exit code 25 of plugins
# because they are executed twice and get killed because of using
# the same ressource.
# Sending results (when exit code is 25 ) will be skipped with this
# enabled.
workaround_rc_25=off
