gearmand failed to load module

[scottwilkerson]

Being you have specified
Local server gearmand sysconfig: /etc/sysconfig/gearmand --> OPTIONS="-L <IPADDRESS>"
gearmand is only going to listen on <IPADDRESS> and not localhost

So this...
Local server gearmand: /etc/mod_gearman/mod_gearman_neb.conf --> server=localhost:4730
will need to be changed to
Local server gearmand: /etc/mod_gearman/mod_gearman_neb.conf --> server=<IPADDRESS>:4730

then nagios restarted

Code: Select all

service nagios restart

[emartine]

Active checks stopped after I updated the system using standard RHEL repositories. Reinstalling is not an option so I need to find out what has caused this.

[emartine]

I've change all gearman configuration to have an IP address. Same result with command submitting.

[emartine]

When I restart nagios. Seems like everything is ok.

[1422903127] Caught SIGTERM, shutting down...
[1422903127] Successfully shutdown... (PID=18631)
[1422903127] Event broker module 'NERD' deinitialized successfully.
[1422903127] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' deinitialized successfully.
[1422903127] ndomod: Shutdown complete.
[1422903127] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
[1422903127] Nagios 4.0.8 starting... (PID=19490)
[1422903127] Local time is Mon Feb 02 12:52:07 CST 2015
[1422903127] LOG VERSION: 2.0
[1422903127] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1422903127] qh: core query handler registered
[1422903127] nerd: Channel hostchecks registered successfully
[1422903127] nerd: Channel servicechecks registered successfully
[1422903127] nerd: Channel opathchecks registered successfully
[1422903127] nerd: Fully initialized and ready to rock!
[1422903127] wproc: Successfully registered manager as @wproc with query handler
[1422903127] wproc: Registry request: name=Core Worker 19494;pid=19494
[1422903127] wproc: Registry request: name=Core Worker 19495;pid=19495
[1422903127] wproc: Registry request: name=Core Worker 19493;pid=19493
[1422903127] wproc: Registry request: name=Core Worker 19492;pid=19492
[1422903127] mod_gearman: initialized version 1.5.0b1 (libgearman 1.1.8)
[1422903127] Event broker module '/usr/lib64/mod_gearman/mod_gearman.o' initialized successfully.
[1422903127] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1422903127] ndomod: Successfully connected to data sink. 0 queued items to flush.
[1422903127] ndomod registered for process data
[1422903127] ndomod registered for log data'
[1422903127] ndomod registered for system command data'
[1422903127] ndomod registered for event handler data'
[1422903127] ndomod registered for notification data'
[1422903127] ndomod registered for comment data'
[1422903127] ndomod registered for downtime data'
[1422903127] ndomod registered for flapping data'
[1422903127] ndomod registered for program status data'
[1422903127] ndomod registered for host status data'
[1422903127] ndomod registered for service status data'
[1422903127] ndomod registered for adaptive program data'
[1422903127] ndomod registered for adaptive host data'
[1422903127] ndomod registered for adaptive service data'
[1422903127] ndomod registered for external command data'
[1422903127] ndomod registered for aggregated status data'
[1422903127] ndomod registered for retention data'
[1422903127] ndomod registered for contact data'
[1422903127] ndomod registered for contact notification data'
[1422903127] ndomod registered for acknowledgement data'
[1422903127] ndomod registered for state change data'
[1422903127] ndomod registered for contact status data'
[1422903127] ndomod registered for adaptive contact data'
[1422903127] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1422903127] Successfully launched command file worker with pid 19501

I will restart the system again for good measure.

[tgriep]

Good news! Let us know if the issue is resolved.

[emartine]

After restarting the system I noticed that the engine was not runnning. I restarted that through the web interface. Once everything was good there I logged onto the system and noticed nagios was not running.
I stopped gearmand, stopped the workers (local and remote), started gearmand, started workers and then started nagios. Seems like all is working as it should. I am now able to submit commands again and active checks are going through fine. So I am still on a quest to find out what happened. I would like to know what went wrong. I still can't find any logs that will tell me what the issue was. This definitely doesn't give me the warm and fuzzies for updating the production servers.

[tgriep]

Did you setup the gearman daemons to run at startup when the server is rebooted?

If not, here is how you do that.
First line is for the gearman server and the second line is just for the worker.

Code: Select all

chkconfig gearmand on
chkconfig mod_gearman_worker on

[emartine]

These are all set to come up at startup. I believe these are added as part of the packages. Maybe the order in which they startup is not correct? What should the priority levels be?

mod_gearman_worker 0:off 1:off 2:on 3:on 4:on 5:on 6:off
gearmand 0:off 1:off 2:on 3:on 4:on 5:on 6:off
mysqld 0:off 1:off 2:off 3:on 4:off 5:on 6:off
nagios 0:off 1:off 2:off 3:on 4:off 5:on 6:off
nagiosxi 0:off 1:off 2:off 3:on 4:off 5:on 6:off
ndo2db 0:off 1:off 2:off 3:on 4:off 5:on 6:off

[tgriep]

Here is how it is setup on my server

Code: Select all

gearmand:# chkconfig: - 85 15
mod_gearman_worker:# chkconfig: 2345 85 15
mysqld:# chkconfig: - 64 36
nagios:# chkconfig: 345 99 01
nagiosxi:# chkconfig: 345 99 01
ndo2db:# chkconfig: 345 99 01

The Required start and stop didn't have any special settings.

[emartine]

This is exactly what exists in my init.d scripts as well.