V5.2 Issue: Nagios service dies during Apply Configuration

[rajasegar]

Nagios Apply Configuration was working fine before the upgrade.
After upgrade, the nagios service dies during Apply Configuration.
We need to manually start the process.

We had this problem long time back and support did something to solve the problem.
Now it is back. No error in all the logs.

The other 2 XI servers is fine after the upgrade, maybe because they are not using gearman.

Please assist.

[Box293]

What do you mean by V5.3 ?

Can you please upload the file:
/etc/sudoers
Also, any files in /etc/sudoers.d/

[rajasegar]

What do you mean by V5.3 ?

Can you please upload the file:
/etc/sudoers
Also, any files in /etc/sudoers.d/

Sorry typo, 5.2. This is the same problem you did remote to our server before.
The funny thing is sometimes Apply Configuration works fine.

No files in /etc/sudoers.d

Code: Select all

## Sudoers allows particular users to run various commands as
## the root user, without needing the root password.
##
## Examples are provided at the bottom of the file for collections
## of related commands, which can then be delegated out to particular
## users or groups.
##
## This file must be edited with the 'visudo' command.

## Host Aliases
## Groups of machines. You may prefer to use hostnames (perhaps using
## wildcards for entire domains) or IP addresses instead.
# Host_Alias     FILESERVERS = fs1, fs2
# Host_Alias     MAILSERVERS = smtp, smtp2

## User Aliases
## These aren't often necessary, as you can use regular groups
## (ie, from files, LDAP, NIS, etc) in this file - just use %groupname
## rather than USERALIAS
# User_Alias ADMINS = jsmith, mikem

## Command Aliases
## These are groups of related commands...

## Networking
# Cmnd_Alias NETWORKING = /sbin/route, /sbin/ifconfig, /bin/ping, /sbin/dhclient, /usr/bin/net, /sbin/iptable                                                           s, /usr/bin/rfcomm, /usr/bin/wvdial, /sbin/iwconfig, /sbin/mii-tool

## Installation and management of software
# Cmnd_Alias SOFTWARE = /bin/rpm, /usr/bin/up2date, /usr/bin/yum

## Services
# Cmnd_Alias SERVICES = /sbin/service, /sbin/chkconfig

## Updating the locate database
# Cmnd_Alias LOCATE = /usr/bin/updatedb

## Storage
# Cmnd_Alias STORAGE = /sbin/fdisk, /sbin/sfdisk, /sbin/parted, /sbin/partprobe, /bin/mount, /bin/umount

## Delegating permissions
# Cmnd_Alias DELEGATING = /usr/sbin/visudo, /bin/chown, /bin/chmod, /bin/chgrp

## Processes
# Cmnd_Alias PROCESSES = /bin/nice, /bin/kill, /usr/bin/kill, /usr/bin/killall

## Drivers
# Cmnd_Alias DRIVERS = /sbin/modprobe

# Defaults specification

#
# Disable "ssh hostname sudo <cmd>", because it will show the password in clear.
#         You have to run "ssh -t hostname sudo <cmd>".
#
#####Defaults    requiretty

#
# Refuse to run if unable to disable echo on the tty. This setting should also be
# changed in order to be able to use sudo without a tty. See requiretty above.
#
Defaults   !visiblepw

#
# Preserving HOME has security implications since many programs
# use it when searching for configuration files. Note that HOME
# is already set when the the env_reset option is enabled, so
# this option is only effective for configurations where either
# env_reset is disabled or HOME is present in the env_keep list.
my02390 ALL=(ALL)       NOPASSWD: ALL
#nagios  ALL=(ALL)       NOPASSWD: ALL
#
Defaults    always_set_home

Defaults    env_reset
Defaults    env_keep =  "COLORS DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS"
Defaults    env_keep += "MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE"
Defaults    env_keep += "LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES"
Defaults    env_keep += "LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE"
Defaults    env_keep += "LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY"
Defaults    env_keep += "TZ"

#
# Adding HOME to env_keep may enable a user to run unrestricted
# commands via sudo.
#
# Defaults   env_keep += "HOME"

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin

## Next comes the main part: which users can run what software on
## which machines (the sudoers file can be shared between multiple
## systems).
## Syntax:
##
##      user    MACHINE=COMMANDS
##
## The COMMANDS section may have other options added to it.
##
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL

## Allows members of the 'sys' group to run networking, software,
## service management apps and more.
# %sys ALL = NETWORKING, SOFTWARE, SERVICES, STORAGE, DELEGATING, PROCESSES, LOCATE, DRIVERS

## Allows people in group wheel to run all commands
# %wheel        ALL=(ALL)       ALL

## Same thing without a password
# %wheel        ALL=(ALL)       NOPASSWD: ALL

## Allows members of the users group to mount and unmount the
## cdrom as root
# %users  ALL=/sbin/mount /mnt/cdrom, /sbin/umount /mnt/cdrom

## Allows members of the users group to shutdown this system
# %users  localhost=/sbin/shutdown -h now

## Read drop-in files from /etc/sudoers.d (the # here does not mean a comment)
#includedir /etc/sudoers.d




User_Alias      NAGIOSXI=nagios
User_Alias              NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autod                                                           iscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/au                                                           todiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *

[tgriep]

Can you run the following tail command, click on Apply Config in XI and post the output of the tail?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

[rajasegar]

Can you run the following tail command, click on Apply Config in XI and post the output of the tail?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

See below. I had to start Nagios service manually.

Code: Select all

[nagios@nagiosprodxi1 ~]$ tail -f /usr/local/nagiosxi/var/cmdsubsys.log

PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-10-13 06:59:21--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.loginâ



2015-10-13 06:59:22 (1002 KB/s) - ânagiosql.loginâ

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=nagiosqldb' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-10-13 06:59:23--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.loginâ



2015-10-13 06:59:23 (561 KB/s) - ânagiosql.loginâ

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=nagiosqldb' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-10-13 06:59:24--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.export.monitoringâ



2015-10-13 06:59:39 (952 B/s) - ânagiosql.export.monitoringâ

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 9809 services.
        Checked 1255 hosts.
        Checked 254 host groups.
        Checked 11 service groups.
        Checked 197 contacts.
        Checked 62 contact groups.
        Checked 192 commands.
        Checked 89 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 1255 hosts
        Checked 522 service dependencies
        Checked 0 host dependencies
        Checked 89 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...done.
Stopping nagios: .done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated

PROCESSED 1 COMMANDS
^C
You have new mail in /var/spool/mail/nagios
[nagios@nagiosprodxi1 ~]$ service nagios status
nagios is not running
[nagios@nagiosprodxi1 ~]$ service nagios start
Starting nagios: done.
[nagios@nagiosprodxi1 ~]$

[Box293]

Is there anything recorded in /usr/local/nagios/var/nagios.log to indicate why it stopped after starting?

As a separate test, can you do the following:

Code: Select all

tail -f /usr/local/nagios/var/nagios.log

Then go to Admin > System Information > System Status
Click the gear icon for the Monitoring Engine and select restart

Does nagios correctly stop and start, what appears in the tail output?

[rajasegar]

Is there anything recorded in /usr/local/nagios/var/nagios.log to indicate why it stopped after starting?

As a separate test, can you do the following:

Code: Select all

tail -f /usr/local/nagios/var/nagios.log

Then go to Admin > System Information > System Status
Click the gear icon for the Monitoring Engine and select restart

Does nagios correctly stop and start, what appears in the tail output?

Previous log extract.

nagios_exp.log

Restart via Monitoring Engine. It did not start up. Had to start using another session via command line.

Code: Select all

[nagios@nagiosprodxi1 var]$ tail -f nagios.log
tail: inotify cannot be used, reverting to polling
[1444691709] SERVICE ALERT: SGR_BR_KompleksPenchala;Port 18 Status - Serial0/0/0;CRITICAL;SOFT;1;CRITICAL - snmpwalk returns no product name (sysDescr), wrong device
[1444691719] SERVICE ALERT: NET_STQ_bhqkpkf521107;Memory;CRITICAL;SOFT;1;CRITICAL - could not contact snmp agent, wrong device
[1444691759] SERVICE ALERT: NET_MSB_VFW5;CPU;OK;SOFT;2;OK - cpu noSuchInstance usage (5 min avg.) is 0.00%
[1444691759] SERVICE ALERT: SGR_BR_KompleksPenchala;Port 18 Status - Serial0/0/0;OK;SOFT;2;OK - Serial0/0/0 is up/up
[1444691769] SERVICE ALERT: CIMBMSMTP02;CPU Load;OK;SOFT;2;OK - load average: 0.51, 0.54, 0.51
[1444691769] SERVICE ALERT: MY1PLPRDMFT01;CPU Load;WARNING;SOFT;1;WARNING - load average: 0.93, 0.93, 0.77
[1444691779] SERVICE ALERT: MY1PIPP1;AS400 CPU Load;OK;HARD;3;OK - CPU Load (56.7%)
[1444691789] SERVICE ALERT: NET_STQ_BKPKF52061;Memory;OK;SOFT;2;OK - tmm memory usage is 5.18%, host memory usage is 98.81%
[1444691799] SERVICE ALERT: Cambodia_UAT_SIT_DR;AS400 Nagios Login Alt;CRITICAL;SOFT;5;CRITICAL: Network error:java.net.SocketException: Connection reset
[1444691859] SERVICE ALERT: GTWPRD-APP;File:  IFSSC FILE OUT;CRITICAL;SOFT;1;CRITICAL: File still exist in Outgoing Folder. Please takes necessary actions.
[1444691869] SERVICE ALERT: NET_MSB_REVERSE PROXY PRIMARY;CPU;OK;SOFT;2;OK - cpu 1 usage is 3.00%
[1444691870] Caught SIGTERM, shutting down...
[1444691870] Successfully shutdown... (PID=12407)
[1444691870] Event broker module 'NERD' deinitialized successfully.
[1444691870] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' deinitialized successfully.
[1444691870] ndomod: Shutdown complete.
[1444691870] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
[1444691871] Nagios 4.1.1 starting... (PID=1322)
[1444691871] Local time is Tue Oct 13 07:17:51 MYT 2015
[1444691871] LOG VERSION: 2.0
[1444691871] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1444691871] qh: core query handler registered
[1444691871] nerd: Channel hostchecks registered successfully
[1444691871] nerd: Channel servicechecks registered successfully
[1444691871] nerd: Channel opathchecks registered successfully
[1444691871] nerd: Fully initialized and ready to rock!
[1444691871] wproc: Successfully registered manager as @wproc with query handler
[1444691871] wproc: Registry request: name=Core Worker 1324;pid=1324
[1444691871] wproc: Registry request: name=Core Worker 1332;pid=1332
[1444691871] wproc: Registry request: name=Core Worker 1326;pid=1326
[1444691871] wproc: Registry request: name=Core Worker 1330;pid=1330
[1444691871] wproc: Registry request: name=Core Worker 1329;pid=1329
[1444691871] wproc: Registry request: name=Core Worker 1328;pid=1328
[1444691871] wproc: Registry request: name=Core Worker 1338;pid=1338
[1444691871] wproc: Registry request: name=Core Worker 1333;pid=1333
[1444691871] wproc: Registry request: name=Core Worker 1334;pid=1334
[1444691871] wproc: Registry request: name=Core Worker 1335;pid=1335
[1444691871] wproc: Registry request: name=Core Worker 1336;pid=1336
[1444691871] wproc: Registry request: name=Core Worker 1337;pid=1337
[1444691871] wproc: Registry request: name=Core Worker 1343;pid=1343
[1444691871] wproc: Registry request: name=Core Worker 1341;pid=1341
[1444691871] wproc: Registry request: name=Core Worker 1339;pid=1339
[1444691871] wproc: Registry request: name=Core Worker 1356;pid=1356
[1444691871] wproc: Registry request: name=Core Worker 1342;pid=1342
[1444691871] wproc: Registry request: name=Core Worker 1348;pid=1348
[1444691871] wproc: Registry request: name=Core Worker 1345;pid=1345
[1444691871] wproc: Registry request: name=Core Worker 1346;pid=1346
[1444691871] wproc: Registry request: name=Core Worker 1344;pid=1344
[1444691871] wproc: Registry request: name=Core Worker 1347;pid=1347
[1444691871] wproc: Registry request: name=Core Worker 1352;pid=1352
[1444691871] wproc: Registry request: name=Core Worker 1349;pid=1349
[1444691871] wproc: Registry request: name=Core Worker 1350;pid=1350
[1444691871] wproc: Registry request: name=Core Worker 1351;pid=1351
[1444691871] wproc: Registry request: name=Core Worker 1355;pid=1355
[1444691871] wproc: Registry request: name=Core Worker 1354;pid=1354
[1444691871] wproc: Registry request: name=Core Worker 1353;pid=1353
[1444691871] wproc: Registry request: name=Core Worker 1340;pid=1340
[1444691871] mod_gearman: initialized version 1.5.0b1 (libgearman 1.1.8)
[1444691871] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' initialized successfully.
[1444691871] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1444691871] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1444691871] ndomod registered for process data
[1444691871] ndomod registered for log data'
[1444691871] ndomod registered for system command data'
[1444691871] ndomod registered for event handler data'
[1444691871] ndomod registered for notification data'
[1444691871] ndomod registered for comment data'
[1444691871] ndomod registered for downtime data'
[1444691871] ndomod registered for flapping data'
[1444691871] ndomod registered for program status data'
[1444691871] ndomod registered for host status data'
[1444691871] ndomod registered for service status data'
[1444691871] ndomod registered for adaptive program data'
[1444691871] ndomod registered for adaptive host data'
[1444691871] ndomod registered for adaptive service data'
[1444691871] ndomod registered for external command data'
[1444691871] ndomod registered for aggregated status data'
[1444691871] ndomod registered for retention data'
[1444691871] ndomod registered for contact data'
[1444691871] ndomod registered for contact notification data'
[1444691871] ndomod registered for acknowledgement data'
[1444691871] ndomod registered for state change data'
[1444691871] ndomod registered for contact status data'
[1444691871] ndomod registered for adaptive contact data'
[1444691871] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1444691874] Successfully launched command file worker with pid 1359
[1444691917] Nagios 4.1.1 starting... (PID=1709)
[1444691917] Local time is Tue Oct 13 07:18:37 MYT 2015
[1444691917] LOG VERSION: 2.0
[1444691917] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1444691917] qh: core query handler registered
[1444691917] nerd: Channel hostchecks registered successfully
[1444691917] nerd: Channel servicechecks registered successfully
[1444691917] nerd: Channel opathchecks registered successfully
[1444691917] nerd: Fully initialized and ready to rock!
[1444691917] wproc: Successfully registered manager as @wproc with query handler
[1444691917] wproc: Registry request: name=Core Worker 1712;pid=1712
[1444691917] wproc: Registry request: name=Core Worker 1711;pid=1711
[1444691917] wproc: Registry request: name=Core Worker 1714;pid=1714
[1444691917] wproc: Registry request: name=Core Worker 1713;pid=1713
[1444691917] wproc: Registry request: name=Core Worker 1718;pid=1718
[1444691917] wproc: Registry request: name=Core Worker 1720;pid=1720
[1444691917] wproc: Registry request: name=Core Worker 1737;pid=1737
[1444691917] wproc: Registry request: name=Core Worker 1736;pid=1736
[1444691917] wproc: Registry request: name=Core Worker 1730;pid=1730
[1444691917] wproc: Registry request: name=Core Worker 1725;pid=1725
[1444691917] wproc: Registry request: name=Core Worker 1727;pid=1727
[1444691917] wproc: Registry request: name=Core Worker 1728;pid=1728
[1444691917] wproc: Registry request: name=Core Worker 1715;pid=1715
[1444691917] wproc: Registry request: name=Core Worker 1716;pid=1716
[1444691917] wproc: Registry request: name=Core Worker 1729;pid=1729
[1444691917] wproc: Registry request: name=Core Worker 1719;pid=1719
[1444691917] wproc: Registry request: name=Core Worker 1734;pid=1734
[1444691917] wproc: Registry request: name=Core Worker 1724;pid=1724
[1444691917] wproc: Registry request: name=Core Worker 1731;pid=1731
[1444691917] wproc: Registry request: name=Core Worker 1726;pid=1726
[1444691917] wproc: Registry request: name=Core Worker 1722;pid=1722
[1444691917] wproc: Registry request: name=Core Worker 1735;pid=1735
[1444691917] wproc: Registry request: name=Core Worker 1738;pid=1738
[1444691917] wproc: Registry request: name=Core Worker 1741;pid=1741
[1444691917] wproc: Registry request: name=Core Worker 1721;pid=1721
[1444691917] wproc: Registry request: name=Core Worker 1740;pid=1740
[1444691917] wproc: Registry request: name=Core Worker 1733;pid=1733
[1444691917] wproc: Registry request: name=Core Worker 1723;pid=1723
[1444691917] wproc: Registry request: name=Core Worker 1739;pid=1739
[1444691917] wproc: Registry request: name=Core Worker 1732;pid=1732
[1444691917] mod_gearman: initialized version 1.5.0b1 (libgearman 1.1.8)
[1444691917] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' initialized successfully.
[1444691917] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1444691917] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1444691917] ndomod registered for process data
[1444691917] ndomod registered for log data'
[1444691917] ndomod registered for system command data'
[1444691917] ndomod registered for event handler data'
[1444691917] ndomod registered for notification data'
[1444691917] ndomod registered for comment data'
[1444691917] ndomod registered for downtime data'
[1444691917] ndomod registered for flapping data'
[1444691917] ndomod registered for program status data'
[1444691917] ndomod registered for host status data'
[1444691917] ndomod registered for service status data'
[1444691917] ndomod registered for adaptive program data'
[1444691917] ndomod registered for adaptive host data'
[1444691917] ndomod registered for adaptive service data'
[1444691917] ndomod registered for external command data'
[1444691917] ndomod registered for aggregated status data'
[1444691917] ndomod registered for retention data'
[1444691917] ndomod registered for contact data'
[1444691917] ndomod registered for contact notification data'
[1444691917] ndomod registered for acknowledgement data'
[1444691917] ndomod registered for state change data'
[1444691917] ndomod registered for contact status data'
[1444691917] ndomod registered for adaptive contact data'
[1444691917] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1444691920] Successfully launched command file worker with pid 1744

[Box293]

So the next test I think we need to try is to comment out the Mod_Gearman module in nagios.cfg and then do the same steps of restarting the monitoring engine via the Admin menu and tailing the log.

When doing this, does it start and remain started?

[rajasegar]

So the next test I think we need to try is to comment out the Mod_Gearman module in nagios.cfg and then do the same steps of restarting the monitoring engine via the Admin menu and tailing the log.

When doing this, does it start and remain started?

It works fine now when commented.

Code: Select all

[1444694624] Caught SIGTERM, shutting down...
[1444694624] Successfully shutdown... (PID=1709)
[1444694624] Event broker module 'NERD' deinitialized successfully.
[1444694624] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' deinitialized successfully.
[1444694624] ndomod: Shutdown complete.
[1444694624] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
[1444694625] Nagios 4.1.1 starting... (PID=11943)
[1444694625] Local time is Tue Oct 13 08:03:45 MYT 2015
[1444694625] LOG VERSION: 2.0
[1444694625] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1444694625] qh: core query handler registered
[1444694625] nerd: Channel hostchecks registered successfully
[1444694625] nerd: Channel servicechecks registered successfully
[1444694625] nerd: Channel opathchecks registered successfully
[1444694625] nerd: Fully initialized and ready to rock!
[1444694625] wproc: Successfully registered manager as @wproc with query handler
[1444694625] wproc: Registry request: name=Core Worker 11947;pid=11947
[1444694625] wproc: Registry request: name=Core Worker 11946;pid=11946
[1444694625] wproc: Registry request: name=Core Worker 11951;pid=11951
[1444694625] wproc: Registry request: name=Core Worker 11953;pid=11953
[1444694625] wproc: Registry request: name=Core Worker 11957;pid=11957
[1444694625] wproc: Registry request: name=Core Worker 11948;pid=11948
[1444694625] wproc: Registry request: name=Core Worker 11952;pid=11952
[1444694625] wproc: Registry request: name=Core Worker 11958;pid=11958
[1444694625] wproc: Registry request: name=Core Worker 11961;pid=11961
[1444694625] wproc: Registry request: name=Core Worker 11959;pid=11959
[1444694625] wproc: Registry request: name=Core Worker 11960;pid=11960
[1444694625] wproc: Registry request: name=Core Worker 11950;pid=11950
[1444694625] wproc: Registry request: name=Core Worker 11962;pid=11962
[1444694625] wproc: Registry request: name=Core Worker 11955;pid=11955
[1444694625] wproc: Registry request: name=Core Worker 11965;pid=11965
[1444694625] wproc: Registry request: name=Core Worker 11954;pid=11954
[1444694625] wproc: Registry request: name=Core Worker 11973;pid=11973
[1444694625] wproc: Registry request: name=Core Worker 11967;pid=11967
[1444694625] wproc: Registry request: name=Core Worker 11969;pid=11969
[1444694625] wproc: Registry request: name=Core Worker 11975;pid=11975
[1444694625] wproc: Registry request: name=Core Worker 11971;pid=11971
[1444694625] wproc: Registry request: name=Core Worker 11972;pid=11972
[1444694625] wproc: Registry request: name=Core Worker 11966;pid=11966
[1444694625] wproc: Registry request: name=Core Worker 11976;pid=11976
[1444694625] wproc: Registry request: name=Core Worker 11978;pid=11978
[1444694625] wproc: Registry request: name=Core Worker 11977;pid=11977
[1444694625] wproc: Registry request: name=Core Worker 11980;pid=11980
[1444694625] wproc: Registry request: name=Core Worker 11968;pid=11968
[1444694625] wproc: Registry request: name=Core Worker 11963;pid=11963
[1444694625] wproc: Registry request: name=Core Worker 11970;pid=11970
[1444694625] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1444694625] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1444694625] ndomod registered for process data
[1444694625] ndomod registered for log data'
[1444694625] ndomod registered for system command data'
[1444694625] ndomod registered for event handler data'
[1444694625] ndomod registered for notification data'
[1444694625] ndomod registered for comment data'
[1444694625] ndomod registered for downtime data'
[1444694625] ndomod registered for flapping data'
[1444694625] ndomod registered for program status data'
[1444694625] ndomod registered for host status data'
[1444694625] ndomod registered for service status data'
[1444694625] ndomod registered for adaptive program data'
[1444694625] ndomod registered for adaptive host data'
[1444694625] ndomod registered for adaptive service data'
[1444694625] ndomod registered for external command data'
[1444694625] ndomod registered for aggregated status data'
[1444694625] ndomod registered for retention data'
[1444694625] ndomod registered for contact data'
[1444694625] ndomod registered for contact notification data'
[1444694625] ndomod registered for acknowledgement data'
[1444694625] ndomod registered for state change data'
[1444694625] ndomod registered for contact status data'
[1444694625] ndomod registered for adaptive contact data'
[1444694625] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1444694628] Successfully launched command file worker with pid 16077
[1444694651] SERVICE ALERT: esbwmbdr02;CPU Load;CRITICAL;SOFT;2;CRITICAL - load average: 4.22, 2.38, 1.59

[ssax]

rajasegar, we have our developers looking into this as we now have multiple people reporting the same issue. We've reached out to the mod_gearman developers as well and they haven't heard any reports of this.