Nagios XI host check orphaned and duplicate nagios process

[emartine]

###############################################################################
#
# Mod-Gearman - distribute checks with gearman
#
# Copyright (c) 2010 Sven Nierlein
#
# Mod-Gearman NEB Module Config
#
###############################################################################

# use debug to increase the verbosity of the module.
# Possible values are:
# 0 = only errors
# 1 = debug messages
# 2 = trace messages
# 3 = trace and all gearman related logs are going to stdout.
# Default is 0.
debug=0

# Path to the logfile.
logfile=/var/log/mod_gearman2/mod_gearman_neb.log

# sets the addess of your gearman job server. Can be specified
# more than once to add more server.
server=localhost:4730


# sets the address of your 2nd (duplicate) gearman job server. Can
# be specified more than once o add more servers.
#dupserver=<host>:<port>


# defines if the module should distribute execution of
# eventhandlers.
eventhandler=yes


# defines if the module should distribute execution of
# service checks.
services=yes


# defines if the module should distribute execution of
# host checks.
hosts=yes


# sets a list of hostgroups which will go into seperate
# queues. Either specify a comma seperated list or use
# multiple lines.
#hostgroups=name1
#hostgroups=name2,name3


# sets a list of servicegroups which will go into seperate
# queues.
#servicegroups=name1,name2,name3

# Set this to 'no' if you want Mod-Gearman to only take care of
# servicechecks. No hostchecks will be processed by Mod-Gearman. Use
# this option to disable hostchecks and still have the possibility to
# use hostgroups for easy configuration of your services.
# If set to yes, you still have to define which hostchecks should be
# processed by either using 'hosts' or the 'hostgroups' option.
# Default is Yes.
do_hostchecks=yes
# This settings determines if all eventhandlers go into a single
# 'eventhandlers' queue or into the same queue like normal checks
# would do.
route_eventhandler_like_checks=no

# enables or disables encryption. It is strongly
# advised to not disable encryption. Anybody will be
# able to inject packages to your worker.
# Encryption is enabled by default and you have to
# explicitly disable it.
# When using encryption, you will either have to
# specify a shared password with key=... or a
# keyfile with keyfile=...
# Default is On.
encryption=yes


# A shared password which will be used for
# encryption of data pakets. Should be at least 8
# bytes long. Maximum length is 32 characters.
key=should_be_changed


# The shared password will be read from this file.
# Use either key or keyfile. Only the first 32
# characters will be used.
#keyfile=/path/to/secret.file


# use_uniq_jobs
# Using uniq keys prevents the gearman queues from filling up when there
# is no worker. However, gearmand seems to have problems with the uniq
# key and sometimes jobs get stuck in the queue. Set this option to 'off'
# when you run into problems with stuck jobs but make sure your worker
# are running.
use_uniq_jobs=on



###############################################################################
#
# NEB Module Config
#
# the following settings are for the neb module only and
# will be ignored by the worker.
#
###############################################################################

# sets a list of hostgroups which will not be executed
# by gearman. They are just passed through.
# Default is none
localhostgroups=


# sets a list of servicegroups which will not be executed
# by gearman. They are just passed through.
# Default is none
localservicegroups=

# The queue_custom_variable can be used to define the target queue
# by a custom variable in addition to host/servicegroups. When set
# for ex. to 'WORKER' you then could define a '_WORKER' custom
# variable for your hosts and services to directly set the worker
# queue. The host queue is inherited unless overwritten
# by a service custom variable. Set the value of your custom
# variable to 'local' to bypass Mod-Gearman (Same behaviour as in
# localhostgroups/localservicegroups).
#queue_custom_variable=WORKER

# Number of result worker threads. Usually one is
# enough. You may increase the value if your
# result queue is not processed fast enough.
# Default: 1
result_workers=1


# defines if the module should distribute perfdata
# to gearman.
# Note: processing of perfdata is not part of
# mod_gearman. You will need additional worker for
# handling performance data. For example: pnp4nagios
# Performance data is just written to the gearman
# queue.
# Default: no
perfdata=no

# perfdata mode overwrite helps preventing the perdata queue getting to big
# 1 = overwrote
# 2 = append
perfdata_mode=1

# The Mod-Gearman NEB module will submit a fake result for orphaned host
# checks with a message saying there is no worker running for this
# queue. Use this option to get better reporting results, otherwise your
# hosts will keep their last state as long as there is no worker
# running.
# Default: yes
orphan_host_checks=yes

# Same like 'orphan_host_checks' but for services.
# Default: yes
orphan_service_checks=yes

# When accept_clear_results is enabled, the NEB module will accept unencrypted
# results too. This is quite useful if you have lots of passive checks and make
# use of send_gearman/send_multi where you would have to spread the shared key to
# all clients using these tools.
# Default is no.
accept_clear_results=no






I don't see anything in the logs that tell me what the issue is but when I submit a command through the NagiosXI site I get a time wait ..... then this screenshot. The process did not execute since service modification did not occur.

[tgriep]

Please attach these files to this post so we can review them.

Code: Select all

/etc/httpd/conf/httpd.conf
/etc/httpd/conf.d/nagiosxi.conf
/etc/httpd/conf.d/ssl.conf

Can you login to the XI server as root and run this tail command.

Code: Select all

tail -f /var/log/httpd/*_log

Then try submitting a command and post any errors from the tail command.
Are you getting any errors if you try and Apply a Configuration?

[emartine]

Files attached

Note that the nagios server IP has been replaced with <HOST IP> and client host name has been replaced with Serverxyz
Will attache 3 at a time since 3 is the attachment limit.

[emartine]

Two more files attached.

[emartine]

Note that the apply configuration command will never finish.

[tgriep]

I see some LDAP authenication settings in the bottom of the httpd.conf file.
Are they valid?

Login to the XI server and run the following to debug the Apply Configuration not finishing. Please post the output.

Code: Select all

cd /usr/local/nagiosxi/scripts
su nagios
./reconfigure_nagios.sh

[emartine]

LDAP AUth settings at the bottom were changed for privacy purposes but are valid.

[emartine]

reconfigure output attached note that password if any was removed.

[ssax]

The apply config is completing but not updating the web UI, the only times I've seen this have been when SSL/TLS Ciphers/Protocols had been changed in the apache configs, have they?

What is the output of this command:

Code: Select all

chage -l nagios

[emartine]

Yes it was! /usr/local/nagiosxi/html/includes/utilsx.inc.php had to be modified again... there was an update not long ago wasn't there?

It is working once again it seems.Hopefully this new version of gearman doesn't experience the same orphaned issue as the previous one.