Monitoring Engine: process No data, Event Queue No output

[xlin125]

This is on a Nagios XI 5.4.8 on Redhat 7.4.

Normally, on the Nagios XI Web Interface, we see the following 6 indicators for XI System Component Status and monitoring engine process status in green:
Monitoring Engine
Performance Grapher
Database Backend
Active Host Checks
Active Service Checks
Notifications

The Apache server RPM package is installed, and apache daemon "httpd" is started as root by default. I reconfigured the apache server to be started as apache by changing the file ownership and file permission for a few apache related files to address the security concern (httpd runs as root). Then I logged into the Nagios XI Web Interface. I noticed that 3 monitoring engine processes (Active Host Checks, Active Service Checks, Notifications) “disappear" (no longer listing), and other 3 still exist showing green. I selected Admin->System Information->Monitoring Engine Status. Under the "Monitoring Engine Process", it displayed “No data”. And under the "Monitoring Engine Event Queue", it displayed “Error: No output from backend!”. When I clicked "Apply Configuration", I noticed the Apply Configuration never completed. What caused the 3 monitoring engine process Active Host Checks, Active Service Checks, and Notifications not listed and no data/no output"? How can I get them back and listed in green?

[scottwilkerson]

How can I get them back and listed in green?

With all due respect, change back whatever you just changed, and I would grab a current copy of the history in case we need to assist you

Code: Select all

history > /tmp/history.txt

Once permissions and ownership are changed on a system, it can be extremely difficult for a person who didn't make the change to know what happened.

[xlin125]

Thanks @scottwilkerson for the response. The changes to Apache are as shown below to allow the user "apache" to start and access the directories:
chown -R apache:apache /var/log/httpd
chown apache /var/run/httpd
chown apache:apache /usr/sbin/httpd

After making these file/directory ownership and permission changes, the user "apache" is able to start the apache server and httpd runs as apache:
$ ps -ef|grep httpd
apache 46026 1 0 Mar16 ? 00:00:23 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46030 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46031 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46032 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46033 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46034 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start

It does not look like these changes would cause this issue. Thanks!

[scottwilkerson]

Thanks @scottwilkerson for the response. The changes to Apache are as shown below to allow the user "apache" to start and access the directories:
chown -R apache:apache /var/log/httpd
chown apache /var/run/httpd
chown apache:apache /usr/sbin/httpd

After making these file/directory ownership and permission changes, the user "apache" is able to start the apache server and httpd runs as apache:
$ ps -ef|grep httpd
apache 46026 1 0 Mar16 ? 00:00:23 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46030 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46031 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46032 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46033 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start
apache 46034 46026 0 Mar16 ? 00:00:00 /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -k start

It does not look like these changes would cause this issue. Thanks!

This sounds better, Nagios XI expects the apache user to be apache, and as such apache is added to the nagios group. If apache was running as root, some files would be created with root as their owner and then the nagios processes (and crons and others) wouldn't be able to modify them.

Are you still experiencing any issues?

[xlin125]

apache is already added to the nagios group:

$ grep apache /etc/group
apache1800:nagcmd,nagios
nagios32966:apache,nagcmd
nagcmd32965:apache,nagios

$ grep nagios /etc/group
apache1800:nagcmd,nagios
nagios32966:apache,nagcmd
nagcmd32965:apache,nagios

However, we are still experiencing this issue.

[scottwilkerson]

can you run the following and report the results

Code: Select all

rm -f /usr/local/nagiosxi/scripts/reconfigure_nagios.lock
chage -l nagios
grep NAGIOSXI /etc/sudoers
tail -50 /var/log/cron

[xlin125]

$ chage -l nagios
Last password change : Feb 05, 2018
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : -1
Number of days of warning before password expires : 7

/usr/local/nagiosxi/scripts/reconfigure_nagios.lock does not exist.

I've asked SA to run the following two commands and send me the results. I do not have read permission on these two files. Once I have the results, I will post them here.
grep NAGIOSXI /etc/sudoers
tail -50 /var/log/cron

[scottwilkerson]

I've asked SA to run the following two commands and send me the results. I do not have read permission on these two files. Once I have the results, I will post them here.
grep NAGIOSXI /etc/sudoers
tail -50 /var/log/cron

thanks

[xlin125]

Please take a look at the outputs requested below. Thanks!

# grep NAGIOSXI /etc/sudoers
User_Alias NAGIOSXI=nagios
User_Alias NAGIOSXIWEB=apache
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/nagios checkconfig
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/ndo2db status
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd start
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd stop
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd restart
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd reload
NAGIOSXI ALL = NOPASSWD:/etc/init.d/npcd status
NAGIOSXI ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/upgrade_to_latest.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/change_timezone.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/reset_config_perms.sh
NAGIOSXI ALL = NOPASSWD:/usr/local/nagiosxi/scripts/backup_xi.sh *
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/messages
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/httpd/error_log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/tail -100 /var/log/mysqld.log
NAGIOSXIWEB ALL = NOPASSWD:/usr/bin/php /usr/local/nagiosxi/html/includes/components/autodiscovery/scripts/autodiscover_new.php *
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
NAGIOSXIWEB ALL = NOPASSWD:/etc/init.d/snmptt restart
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/repair_databases.sh
NAGIOSXIWEB ALL = NOPASSWD:/usr/local/nagiosxi/scripts/manage_services.sh *
#

# tail -50 /var/log/cron
Apr 9 04:23:08 iracavcvsam02 CROND[43108]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:23:09 iracavcvsam02 CROND[43114]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr 9 04:23:09 iracavcvsam02 CROND[43154]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:24:04 iracavcvsam02 CROND[43316]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr 9 04:24:08 iracavcvsam02 CROND[43327]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:24:08 iracavcvsam02 CROND[43328]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr 9 04:24:08 iracavcvsam02 CROND[43352]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr 9 04:24:08 iracavcvsam02 CROND[43382]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr 9 04:24:09 iracavcvsam02 CROND[43422]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr 9 04:24:09 iracavcvsam02 CROND[43428]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Apr 9 04:24:09 iracavcvsam02 CROND[43434]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr 9 04:24:10 iracavcvsam02 CROND[43440]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:25:08 iracavcvsam02 CROND[43601]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr 9 04:25:08 iracavcvsam02 CROND[43600]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Apr 9 04:25:08 iracavcvsam02 CROND[43636]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr 9 04:25:09 iracavcvsam02 CROND[43642]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr 9 04:25:09 iracavcvsam02 CROND[43651]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:25:09 iracavcvsam02 CROND[43659]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr 9 04:25:09 iracavcvsam02 CROND[43676]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr 9 04:25:10 iracavcvsam02 CROND[43682]: (root) CMD (LANG=C LC_ALL=C /usr/bin/mrtg /etc/mrtg/mrtg.cfg --lock-file /var/lock/mrtg/mrtg_l --confcache-file /var/lib/mrtg/mrtg.ok)
Apr 9 04:25:10 iracavcvsam02 CROND[43683]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:25:10 iracavcvsam02 CROND[43689]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php >> /usr/local/nagiosxi/var/dbmaint.log 2>&1)
Apr 9 04:25:10 iracavcvsam02 CROND[43695]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr 9 04:26:05 iracavcvsam02 CROND[43903]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr 9 04:26:08 iracavcvsam02 CROND[43920]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Apr 9 04:26:08 iracavcvsam02 CROND[43921]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr 9 04:26:08 iracavcvsam02 CROND[43941]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:26:09 iracavcvsam02 CROND[43947]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr 9 04:26:09 iracavcvsam02 CROND[43954]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr 9 04:26:09 iracavcvsam02 CROND[43960]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:26:10 iracavcvsam02 CROND[43966]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr 9 04:26:10 iracavcvsam02 CROND[43996]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr 9 04:27:02 iracavcvsam02 CROND[44215]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr 9 04:27:07 iracavcvsam02 CROND[44227]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr 9 04:27:07 iracavcvsam02 CROND[44228]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:27:08 iracavcvsam02 CROND[44241]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr 9 04:27:08 iracavcvsam02 CROND[44271]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
Apr 9 04:27:08 iracavcvsam02 CROND[44312]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr 9 04:27:09 iracavcvsam02 CROND[44318]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr 9 04:27:09 iracavcvsam02 CROND[44336]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:27:09 iracavcvsam02 CROND[44342]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Apr 9 04:28:04 iracavcvsam02 CROND[44557]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php >> /usr/local/nagiosxi/var/cleaner.log 2>&1)
Apr 9 04:28:08 iracavcvsam02 CROND[44592]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php >> /usr/local/nagiosxi/var/perfdataproc.log 2>&1)
Apr 9 04:28:08 iracavcvsam02 CROND[44593]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php >> /usr/local/nagiosxi/var/feedproc.log 2>&1)
Apr 9 04:28:08 iracavcvsam02 CROND[44618]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php >> /usr/local/nagiosxi/var/event_handler.log 2>&1)
Apr 9 04:28:09 iracavcvsam02 CROND[44625]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php >> /usr/local/nagiosxi/var/eventman.log 2>&1)
Apr 9 04:28:09 iracavcvsam02 CROND[44632]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php >> /usr/local/nagiosxi/var/reportengine.log 2>&1)
Apr 9 04:28:09 iracavcvsam02 CROND[44638]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/nom.php >> /usr/local/nagiosxi/var/nom.log 2>&1)
Apr 9 04:28:09 iracavcvsam02 CROND[44644]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php >> /usr/local/nagiosxi/var/cmdsubsys.log 2>&1)
Apr 9 04:28:10 iracavcvsam02 CROND[44650]: (nagios) CMD (/usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php >> /usr/local/nagiosxi/var/sysstat.log 2>&1)
#

[scottwilkerson]

PM sent