Apply Configuration never completes,XI System Components red

[xlin125]

We have two Nagios XI 5.2.3 installed on Redhat 7.3. They had been working fine until last week. I added a new host, then I noticed the Apply Configuration never completed. The workaround is to run reconfigure_nagios.sh, then the host and the Nagios services assigned to this host are added and saved. I found a lot of discussion on how to fix this issue in this forum, and I also read the "Nagios XI - Apply Configuration Never Completes" as suggested. But they are not applied to my case (no luck).

In addition, under XI Admin (Web), I noticed that the following XI System Components have not been running (red with "!") since 9/31/2017 (6 days ago):
Database Maintenance
Command Subsystem
Event Manager
Feed Processor
Report Engine
Cleaner
Nonstop Operations Manager
System Statistics

Only three XI System Component Monitoring Engine, Performance Grapher, and Database Backend are running. The output from running reconfigure_nagios.sh looks fine.

Are these two issues related? How to restart these 8 XI system components? How to resolve this "Apply Configuration never completes"? Thanks!

[tacolover101]

In addition, under XI Admin (Web), I noticed that the following XI System Components have not been running (red with "!") since 9/31/2017 (6 days ago):
Database Maintenance
Command Subsystem
Event Manager
Feed Processor
Report Engine
Cleaner
Nonstop Operations Manager
System Statistics

if all of these services are running properly (to name a few - nagios, mariadb, crond), and it's displaying incorrectly, then I believe there are permissions messed up somewhere.

as with what you mentioned as your previous issue, it sounds like permissions as well. you'd likely need to change them back to what they were initially set to by the Nagios software.

[dwhitfield]

Thanks @tacolover101!

OP, please see https://support.nagios.com/kb/article.php?id=34

Also, what is the output of service crond status? Please put the output in a code block. The "Code" button is the fifth from the left on the post input screen (between Quote and List).

[xlin125]

Thanks @tacolover101!
If there is permission issue, can you tell me which files got messed up with the permissions? Also, what specific services you referred to that may cause these XI System Components not running? It looks like nagios and crond are running fine. I do not see mariadb. Thanks again!

Thanks @dwhitfield!
Here is the output of checking the crond status:
/bin/systemctl status crond
● crond.service - Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-06-21 16:21:05 UTC; 2 months 16 days ago
Main PID: 3322 (crond)
CGroup: /system.slice/crond.service
└─3322 /usr/sbin/crond -n

[scottwilkerson]

let's verify that the nagios user didn't expire

Code: Select all

chage -l nagios

and all the groups look ok,

Code: Select all

cat /etc/group|grep nag

[xlin125]

Thanks @scottwilkerson!

Here are the output:
chage -l nagios
Last password change : Nov 01, 2016
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7

cat /etc/group |grep nag
apache1800:nagios
nagios32966:
nagcmd32965:
nagsupt491596:

[scottwilkerson]

Ok, that looks normal.

Can you also run the following and post the results

Code: Select all

sestatus
tail -50 /var/log/cron
ps -ef|grep cron|grep nagiosxi

Also, please attach the contents of the following file
/etc/cron.d/nagiosxi

[xlin125]

I do not have root on this system, so I can not view the /var/log/cron file.

sestatus
SELinux status: disabled

ps -ef|grep cron |grep nagiosxi
(blank)

Also, we install Nagios XI 5.2.3 software to different directories instead of /usr/local. So /etc/cron.d/nagiosxi does not exist. However, I can show the cron jobs defined for nagios on this Nagios XI 5.2.3 server:
crontab -l
##### PLATFORM_MERGECRON BEGIN #####
##### DO NOT ADD ANY CRON ENTRIES BETWEEN LINES BEGINNING WITH ##### PLATFORM_MERGECRON
#15,45 * * * * /opt/app/nagios/store/backups/scripts/automysqlbackup
0 7 * * * /opt/app/nagios/store/backups/scripts/automysqlbackup
01 * * * * /opt/app/nagios/nagiosxi/cron/recurringdowntime.pl > /opt/app/nagios/nagiosxi/var/recurringdowntime.log 2>&1


I noticed that on this Nagios XI server, there are a lot of cron jobs missing. The following shows the cron jobs defined for nagios on another healthy Nagios XI 5.2.3:
$ crontab -l
##### PLATFORM_MERGECRON BEGIN #####
##### DO NOT ADD ANY CRON ENTRIES BETWEEN LINES BEGINNING WITH ##### PLATFORM_MERGECRON
#15,45 * * * * /opt/app/nagios/store/backups/scripts/automysqlbackup
0 7 * * * /opt/app/nagios/store/backups/scripts/automysqlbackup
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/sysstat.php > /opt/app/nagios/nagiosxi/var/sysstat.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/cmdsubsys.php > /opt/app/nagios/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/eventman.php > /opt/app/nagios/nagiosxi/var/eventman.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/feedproc.php > /opt/app/nagios/nagiosxi/var/feedproc.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/perfdataproc.php > /opt/app/nagios/nagiosxi/var/perfdataproc.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/nom.php > /opt/app/nagios/nagiosxi/var/nom.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/reportengine.php > /opt/app/nagios/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/dbmaint.php > /opt/app/nagios/nagiosxi/var/dbmaint.log 2>&1
* * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/cleaner.php > /opt/app/nagios/nagiosxi/var/cleaner.log 2>&1
01 * * * * /opt/app/nagios/nagiosxi/cron/recurringdowntime.pl > /opt/app/nagios/nagiosxi/var/recurringdowntime.log 2>&1
*/5 * * * * /opt/app/php/bin/php -q /opt/app/nagios/nagiosxi/cron/deadpool.php > /opt/app/nagios/nagiosxi/var/deadpool.log 2>&1
##### PLATFORM_MERGECRON END #####

Does this cause the problem?

[scottwilkerson]

ps -ef|grep cron |grep nagiosxi
(blank)

The fact that this is blank show all the problem, none of the required cron jobs are running on the server as they should be.

Whatever was changed is causing the crons to not run, you will need to consult with an admin that has root access to see why they are not running

[xlin125]

Thanks @scottwilkerson!

I added the missing cron jobs for nagios on this Nagios XI server. All those XI system components are running now. In addition, the Apply Configuration is completed within the expected timeframe.

Thanks to all who help resolve these issues!