Monitoring engine not firing checks and more

GldRush98 · Post by **GldRush98** » Thu Nov 12, 2015 1:40 pm

XI 5.2.1
Here's one I haven't seen before.
I've got an XI system with the following symptoms:
-Monitoring Engine checks are not firing at all
-The controls in the Monitoring Engine Status to stop/start processes show everything green, but they don't do anything when stopped. LIkewise, the Process Start Time seems to be stuck as we have rebooted the machine in an attempt to fix this
-Running through the monitoring wizard, it says it applied successfully but the added host never shows up in XI or CCM anywhere.

Checks are verified not firing by looking at a service in XI and they are all showing a last check time of a couple hours ago.

I'm not sure what they did to the system around that time besides they ran an upgrade from 5.2.0 to 5.2.1. But this could've been not working even before that upgrade was ran, I'm not sure.

I have tried running the repair_databases script - no change.
I have tried manually reapplying configs - no change.

Code: Select all

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Warning: Service 'SftpFaxFileMove' on host 'oddcpuapp03' has no check time period defined!
Warning: Service 'PayerStatus' on host 'vevsp' has no check time period defined!
Warning: Service 'E1processor' on host 'voddcpuapp01' has no check time period defined!
Warning: Service 'Kmart_SoldDate' on host 'voddcpuapp01' has no check time period defined!
Warning: Service 'ShipDate' on host 'voddcpuapp01' has no check time period defined!
        Checked 353 services.
        Checked 107 hosts.
        Checked 15 host groups.
        Checked 26 service groups.
        Checked 12 contacts.
        Checked 8 contact groups.
        Checked 173 commands.
        Checked 11 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 107 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 11 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 5
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:No lock file found in /usr/local/nagios/var/nagios.lock
Starting nagios: done.

Any ideas?

scottwilkerson · Post by **scottwilkerson** » Thu Nov 12, 2015 2:20 pm

Can you run the following

Code: Select all

ps -ef|grep bin/nagios
tail -100 /usr/local/nagios/var/nagios.log

and post the output

GldRush98 · Post by **GldRush98** » Thu Nov 12, 2015 2:44 pm

Sure;

Code: Select all

Linux [root@omnimon2 ~]# ps -ef|grep bin/nagios
nagios   13793     1  0 12:33 ?        00:00:06 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
nagios   13795 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13796 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13797 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13798 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13799 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13800 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13801 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13802 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13803 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13804 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13805 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13806 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13807 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13808 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13809 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13810 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13811 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13812 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios --worker /usr/local/nagios/var/rw/nagios.qh
nagios   13823 13793  0 12:33 ?        00:00:00 /usr/local/nagios/bin/nagios -d /usr/local/nagios/etc/nagios.cfg
root     33011  3113  0 13:43 pts/1    00:00:00 grep bin/nagios
Linux [root@omnimon2 ~]# tail -100 /usr/local/nagios/var/nagios.log
[1447353117] ndomod registered for retention data'
[1447353117] ndomod registered for contact data'
[1447353117] ndomod registered for contact notification data'
[1447353117] ndomod registered for acknowledgement data'
[1447353117] ndomod registered for state change data'
[1447353117] ndomod registered for contact status data'
[1447353117] ndomod registered for adaptive contact data'
[1447353117] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353117] Warning: Service 'SftpFaxFileMove' on host 'oddcpuapp03' has no check time period defined!
[1447353117] Warning: Service 'PayerStatus' on host 'vevsp' has no check time period defined!
[1447353117] Warning: Service 'E1processor' on host 'voddcpuapp01' has no check time period defined!
[1447353117] Warning: Service 'Kmart_SoldDate' on host 'voddcpuapp01' has no check time period defined!
[1447353117] Warning: Service 'ShipDate' on host 'voddcpuapp01' has no check time period defined!
[1447353117] Successfully launched command file worker with pid 12931
[1447353208] Caught SIGTERM, shutting down...
[1447353208] Successfully shutdown... (PID=12901)
[1447353208] Event broker module 'NERD' deinitialized successfully.
[1447353208] ndomod: Shutdown complete.
[1447353208] Event broker module '/usr/local/nagios/bin/ndomod.o' deinitialized successfully.
[1447353210] Nagios 4.1.1 starting... (PID=13793)
[1447353210] Local time is Thu Nov 12 12:33:30 CST 2015
[1447353210] LOG VERSION: 2.0
[1447353210] qh: Socket '/usr/local/nagios/var/rw/nagios.qh' successfully initialized
[1447353210] qh: core query handler registered
[1447353210] nerd: Channel hostchecks registered successfully
[1447353210] nerd: Channel servicechecks registered successfully
[1447353210] nerd: Channel opathchecks registered successfully
[1447353210] nerd: Fully initialized and ready to rock!
[1447353210] wproc: Successfully registered manager as @wproc with query handler
[1447353210] wproc: Registry request: name=Core Worker 13795;pid=13795
[1447353210] wproc: Registry request: name=Core Worker 13796;pid=13796
[1447353210] wproc: Registry request: name=Core Worker 13797;pid=13797
[1447353210] wproc: Registry request: name=Core Worker 13798;pid=13798
[1447353210] wproc: Registry request: name=Core Worker 13799;pid=13799
[1447353210] wproc: Registry request: name=Core Worker 13800;pid=13800
[1447353210] wproc: Registry request: name=Core Worker 13801;pid=13801
[1447353210] wproc: Registry request: name=Core Worker 13802;pid=13802
[1447353210] wproc: Registry request: name=Core Worker 13809;pid=13809
[1447353210] wproc: Registry request: name=Core Worker 13810;pid=13810
[1447353210] wproc: Registry request: name=Core Worker 13805;pid=13805
[1447353210] wproc: Registry request: name=Core Worker 13806;pid=13806
[1447353210] wproc: Registry request: name=Core Worker 13807;pid=13807
[1447353210] wproc: Registry request: name=Core Worker 13808;pid=13808
[1447353210] wproc: Registry request: name=Core Worker 13803;pid=13803
[1447353210] wproc: Registry request: name=Core Worker 13811;pid=13811
[1447353210] wproc: Registry request: name=Core Worker 13812;pid=13812
[1447353210] wproc: Registry request: name=Core Worker 13804;pid=13804
[1447353210] ndomod: NDOMOD 2.0.0 (02-28-2014) Copyright (c) 2009 Nagios Core Development Team and Community Contributors
[1447353210] ndomod: Successfully connected to data sink.  0 queued items to flush.
[1447353210] ndomod registered for process data
[1447353210] ndomod registered for log data'
[1447353210] ndomod registered for system command data'
[1447353210] ndomod registered for event handler data'
[1447353210] ndomod registered for notification data'
[1447353210] ndomod registered for comment data'
[1447353210] ndomod registered for downtime data'
[1447353210] ndomod registered for flapping data'
[1447353210] ndomod registered for program status data'
[1447353210] ndomod registered for host status data'
[1447353210] ndomod registered for service status data'
[1447353210] ndomod registered for adaptive program data'
[1447353210] ndomod registered for adaptive host data'
[1447353210] ndomod registered for adaptive service data'
[1447353210] ndomod registered for external command data'
[1447353210] ndomod registered for aggregated status data'
[1447353210] ndomod registered for retention data'
[1447353210] ndomod registered for contact data'
[1447353210] ndomod registered for contact notification data'
[1447353210] ndomod registered for acknowledgement data'
[1447353210] ndomod registered for state change data'
[1447353210] ndomod registered for contact status data'
[1447353210] ndomod registered for adaptive contact data'
[1447353210] Event broker module '/usr/local/nagios/bin/ndomod.o' initialized successfully.
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] WARNING: Extinfo objects are deprecated and will be removed in future versions
[1447353210] Warning: Service 'SftpFaxFileMove' on host 'oddcpuapp03' has no check time period defined!
[1447353210] Warning: Service 'PayerStatus' on host 'vevsp' has no check time period defined!
[1447353210] Warning: Service 'E1processor' on host 'voddcpuapp01' has no check time period defined!
[1447353210] Warning: Service 'Kmart_SoldDate' on host 'voddcpuapp01' has no check time period defined!
[1447353210] Warning: Service 'ShipDate' on host 'voddcpuapp01' has no check time period defined!
[1447353210] Successfully launched command file worker with pid 13823
[1447356810] Auto-save of retention data completed successfully.
Linux [root@omnimon2 ~]#

scottwilkerson · Post by **scottwilkerson** » Thu Nov 12, 2015 3:13 pm

Lets restart ndo2db

Code: Select all

service ndo2db restart

If that doesn't fix it, can I have you just go to nagios core and see if the host shows up

http://YOUR_SERVER/nagios/

Also, do you have a local of offloaded MySQL DB?

Thanks

GldRush98 · Post by **GldRush98** » Thu Nov 12, 2015 3:28 pm

It looks like the ndo2db restart got the service checks firing again.
I still can't seem to add in hosts via config wizard.

Checked in Nagios Core and I don't see the hosts there either.

Ah, I totally didn't realize it but this system does indeed have an offloaded mysql. I didn't set that up on it... guess it'd be a good time to go through and make sure it was done right?

GldRush98 · Post by **GldRush98** » Thu Nov 12, 2015 3:46 pm

The issue seems to be limited to the Configuration Wizards at this point.

I can manually add in hosts and services via CCM and apply the config and they show up fine. Any wizard I try acts like it's working fine, but never adds anything in.

Post by **Box293** » Thu Nov 12, 2015 10:16 pm

Can you run the following in an ssh session on your Nagios XI host.

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Then go into XI and run a Wizard to completion.

What output is produced in the SSH session?

GldRush98 · Post by **GldRush98** » Fri Nov 13, 2015 10:29 am

Box293 wrote:Can you run the following in an ssh session on your Nagios XI host.
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Then go into XI and run a Wizard to completion.

What output is produced in the SSH session?

Sure, generic network device wizard:

Code: Select all

Linux [root@omnimon2 ~]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
........PROCESSING COMMAND ID 277...
Setting new htaccess credentials
Updating password for user nagiosadmin
OUTPUT=
RETURNCODE=0
...........................



................PROCESSING COMMAND ID 278...
PROCESS COMMAND: CMD=17, DATA=
APPLYING NAGIOSCORE CONFIG...
CMDLINE=cd /usr/local/nagiosxi/scripts && ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-11-13 09:25:54--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.loginâ



2015-11-13 09:25:54 (932 KB/s) - ânagiosql.loginâ

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=foirjqeio3409ji' -O nagiosql.loginLOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost/nagiosxi/includes/components/ccm/
Array
(
    [0] => TEST3.cfg
    [1] => s6iecdka.tmp.cfg
)
IMPORTING /usr/local/nagios/etc/import/TEST3.cfg
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=admin&type=import&importsubmitted=true&chbOverwrite=1&subForm=Import&selImportFile[]=/usr/local/nagios/etc/import/TEST3.cfg' -O nagiosql.import.monitoring
--2015-11-13 09:25:55--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.import.monitoringâ



2015-11-13 09:25:55 (79.5 KB/s) - ânagiosql.import.monitoringâ

IMPORTING /usr/local/nagios/etc/import/s6iecdka.tmp.cfg
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=admin&type=import&importsubmitted=true&chbOverwrite=1&subForm=Import&selImportFile[]=/usr/local/nagios/etc/import/s6iecdka.tmp.cfg' -O nagiosql.import.monitoring
--2015-11-13 09:25:55--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.import.monitoringâ



2015-11-13 09:25:56 (77.7 KB/s) - ânagiosql.import.monitoringâ

RESETTING PERMS
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
--2015-11-13 09:25:57--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.loginâ



2015-11-13 09:25:57 (909 KB/s) - ânagiosql.loginâ

/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=foirjqeio3409ji' -O nagiosql.loginLOGIN SUCCESSFUL!
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2015-11-13 09:25:58--  http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ânagiosql.export.monitoringâ



2015-11-13 09:25:59 (14.9 KB/s) - ânagiosql.export.monitoringâ

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
WARNING: Extinfo objects are deprecated and will be removed in future versions
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
Warning: Service 'SftpFaxFileMove' on host 'oddcpuapp03' has no check time period defined!
Warning: Service 'PayerStatus' on host 'vevsp' has no check time period defined!
Warning: Service 'E1processor' on host 'voddcpuapp01' has no check time period defined!
Warning: Service 'Kmart_SoldDate' on host 'voddcpuapp01' has no check time period defined!
Warning: Service 'ShipDate' on host 'voddcpuapp01' has no check time period defined!
        Checked 353 services.
        Checked 107 hosts.
        Checked 15 host groups.
        Checked 26 service groups.
        Checked 12 contacts.
        Checked 8 contact groups.
        Checked 173 commands.
        Checked 11 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 107 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 11 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 5
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios:. done.
Starting nagios: done.
OUTPUT=Starting nagios: done.
RETURNCODE=0
.
PROCESSED 2 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...^C

scottwilkerson · Post by **scottwilkerson** » Fri Nov 13, 2015 2:22 pm

GldRush98 wrote:
Ah, I totally didn't realize it but this system does indeed have an offloaded mysql. I didn't set that up on it... guess it'd be a good time to go through and make sure it was done right?

If after running that wizard, it doesn't show up in the UI, I would say yes...you need to make sure it is setup correctly

Here is the link to our documentation on that
https://assets.nagios.com/downloads/nag ... Server.pdf

GldRush98 · Post by **GldRush98** » Fri Nov 13, 2015 2:35 pm

I've been over the offloaded config doc and it looks like it was followed properly.

The CCM is working fine, I can add new hosts and services in via it, but any config wizard is not adding in.

Nagios Support Forum

Monitoring engine not firing checks and more

Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more

Re: Monitoring engine not firing checks and more