Monitoring Engine Status shows Process State red--notrunning

[corkyman]

Also,
Monitoring Engine Event Queue shows
Error: No output from backend!

Everything looks good otherwise. Looking at the database table itself shows that daemons are running

Code: Select all

-bash-4.1$ echo 'use nagiosxi; select * from `xi_sysstat` ;' | mysql -unagiosxi -pn@gweb -hmynagiosxidb.pp.tvlport.com  -P3382
sysstat_id      metric  value   update_time
1       dbmaint a:1:{s:10:"last_check";i:1482512402;}   2016-12-23 17:00:02
2       cleaner a:1:{s:10:"last_check";i:1482512581;}   2016-12-23 17:03:01
3       nom     a:1:{s:10:"last_check";i:1482512581;}   2016-12-23 17:03:01
4       reportengine    a:1:{s:10:"last_check";i:1482512581;}   2016-12-23 17:03:01
5       dbbackend       a:5:{s:12:"last_checkin";s:19:"2016-12-23 15:30:25";s:15:"bytes_processed";s:9:"298311141";s:17:"entries_processed";s:6:"612573";s:12:"connect_time";s:19:"2016-12-21 12:36:51";s:15:"disconnect_time";s:19:"0000-00-00 00:00:00";}      2016-12-23 17:03:42
6       daemons a:3:{s:10:"nagioscore";a:4:{s:6:"daemon";s:6:"nagios";s:6:"output";s:31:"nagios (pid 9813) is running...";s:11:"return_code";i:0;s:6:"status";i:0;}s:3:"pnp";a:4:{s:6:"daemon";s:4:"npcd";s:6:"output";s:29:"npcd (pid 2558) is running...";s:11:"return_code";i:0;s:6:"status";i:0;}s:8:"ndoutils";a:4:{s:6:"daemon";s:6:"ndo2db";s:6:"output";s:31:"ndo2db (pid 9745) is running...";s:11:"return_code";i:0;s:6:"status";i:0;}} 2016-12-23 17:03:42
7       nagioscore      a:6:{s:16:"activehostchecks";a:3:{s:4:"1min";s:2:"10";s:4:"5min";s:2:"64";s:5:"15min";s:2:"64";}s:17:"passivehostchecks";a:3:{s:4:"1min";s:1:"0";s:4:"5min";s:1:"0";s:5:"15min";s:1:"0";}s:19:"activeservicechecks";a:3:{s:4:"1min";s:2:"28";s:4:"5min";s:3:"140";s:5:"15min";s:3:"167";}s:20:"passiveservicechecks";a:3:{s:4:"1min";s:1:"0";s:4:"5min";s:1:"0";s:5:"15min";s:1:"0";}s:19:"activehostcheckperf";a:6:{s:11:"min_latency";s:1:"0";s:11:"max_latency";s:7:"1.00063";s:11:"avg_latency";s:18:"0.5132854687499999";s:18:"min_execution_time";s:1:"0";s:18:"max_execution_time";s:7:"0.01535";s:18:"avg_execution_time";s:21:"0.0050332812499999996";}s:22:"activeservicecheckperf";a:6:{s:11:"min_latency";s:1:"0";s:11:"max_latency";s:6:"89.267";s:11:"avg_latency";s:18:"3.5218905693950187";s:18:"min_execution_time";s:6:"0.0026";s:18:"max_execution_time";s:5:"33.96";s:18:"avg_execution_time";s:17:"1.437692206405695";}} 2016-12-23 17:03:42
8       load    a:3:{s:5:"load1";s:4:"0.20";s:5:"load5";s:4:"0.37";s:6:"load15";s:4:"0.35";}    2016-12-23 17:03:42
9       memory  a:6:{s:5:"total";s:4:"3832";s:4:"used";s:4:"2105";s:4:"free";s:4:"1726";s:6:"shared";s:1:"0";s:7:"buffers";s:3:"294";s:6:"cached";s:4:"1018";}   2016-12-23 17:03:42
10      swap    a:3:{s:5:"total";s:4:"2047";s:4:"used";s:1:"0";s:4:"free";s:4:"2047";}  2016-12-23 17:03:42
11      iostat  a:6:{s:4:"user";s:4:"2.07";s:4:"nice";s:4:"0.00";s:6:"system";s:4:"0.10";s:6:"iowait";s:4:"0.00";s:5:"steal";s:4:"0.00";s:4:"idle";s:5:"97.83";} 2016-12-23 17:03:27
12      sysstat a:1:{s:10:"last_check";i:1482512601;}   2016-12-23 17:03:27
13      cmdsubsys       a:1:{s:10:"last_check";i:1482512623;}   2016-12-23 17:03:43
14      feedprocessor   a:1:{s:10:"last_check";i:1482512621;}   2016-12-23 17:03:41
15      perfdataprocessor       a:1:{s:10:"last_check";i:1482512621;}   2016-12-23 17:03:41
16      eventman        a:1:{s:10:"last_check";i:1482512621;}   2016-12-23 17:03:41
17      deadpool_reaper a:1:{s:10:"last_check";i:1482512411;}   2016-12-23 17:00:11
-bash-4.1$

We have two NagiosXI servers - primary and backup. One ends with 051 and another 052. I prefixed the config files uploaded with 051 and 052 respectively. We just migrated our MySQL server from running on each server to the enterprise MySQL server that is accessed by both servers. It feels like the GUI is not picking up the correct status from the data base.

[rkennedy]

Generally this error is mentioned when something isn't offloading / connecting properly. Did you follow this document to do so? https://assets.nagios.com/downloads/nag ... Server.pdf - it looks like you specified 3382 correctly in the dbserver variables so that should be working.

Looking at both ndo2db.cfg files - it appears you have both 051 and 052 using the same exact database. This will not function as expected due to the moving parts in Nagios. If you're looking for a HA setup, I would look at the Linbit solution - https://www.nagios.com/news/2015/10/pre ... nagios-xi/

[corkyman]

I worked with Sean to setup HA. We do not have both servers running ndo2db at the same time so we expect it to work. What else could be going wrong?

[ssax]

The reason why we saw this same issue occur last time was because you had the wrong server set in Admin > System Settings > Program URL (it was pointing to the other server where nagios is not running), you are likely experiencing the same issue, please validate the Program URL and let us know the results.

I don't remember if you have a load balancer in front of the web servers, do you?

[corkyman]

On my primary server 051 the URL is set to http://vhlgnngxi051.tvlport.net/nagiosxi/
We do not have a load balancer.

[rkennedy]

Is that the server that is currently in use, or have you failed over to 052 at this point?

[corkyman]

051 is currently in use; it's our primary. 052 has both ndo2db and nagios services down.

[ssax]

Oh, I remember now, you need to disable the cron jobs as well on the inactive machine, I stated this in the remote that you would need to add the disabling of the cron jobs to your HA scripts.

Basically /etc/cron.d/nagiosxi should only be enabled on the active server, please make sure they are disabled on the inactive server.

Thank you

[corkyman]

Yes, we updated our failover script to disable the cron jobs on a server that is not running Nagios. I checked and we are good. Something else is going on.

[ssax]

Let's move this into a ticket, please email [email protected] with a descriptive subject and detailed body and please include a link back to this thread so that we can get a remote session setup.

Thank you