Apply configuration not completing seems to be hanging

[garethwithers]

Afternoon

We have recently noticed that our apply configuration is not working could someone point me in the direction to troubleshoot this other than here: https://support.nagios.com/kb/article.p ... ategory=44

Have tried all of these with no joy

Kind Regards
Gareth

[mcapra]

Can you share what the tail that KB mentions displays when you run an "Apply Configuration"?

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log

Could you also share a system profile? Either via PM or by posting it as an attachment. From the Nagios XI GUI, you can gather a profile via Admin -> System Profile -> Download Profile.

[garethwithers]

I have included the profile in the ticket and below is the output from the command, there is nothing since a backup was completed yesterday:

Code: Select all

[root@vapngoa101 ~]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
Backing up Nagios XI...
tar: Removing leading `/' from member names

So the apply configuration is not writing to that file

Gareth

[mcapra]

Can you share the outputs of:

Code: Select all

service crond status
service crond restart
#give the system 1-2 minutes to settle
ps aux | grep php

Can you also send over the contents of your apache logs? They're typically located in /var/log/httpd.

[garethwithers]

Hi there

Outputs below:

Code: Select all

[root@vapngoa101 ~]# service crond status
crond (pid  1899) is running...
[root@vapngoa101 ~]# service crond restart
Stopping crond:                                            [  OK  ]
Starting crond:                                            [  OK  ]
[root@vapngoa101 ~]# ps aux | grep php
root     120600  0.0  0.0 103316   832 pts/0    S+   08:37   0:00 grep php

Logs attached

[mcapra]

The issue is that there are PHP scripts that run as cron jobs under the nagios user that are responsible for handing the "Apply Configuration" job (among other things), and they are not running for some reason.

Can you share the outputs of these commands:

Code: Select all

cat /etc/php.ini
php -r 'phpinfo();'
chage -l nagios
cat /etc/passwd | grep nagios
vi /etc/cron.d/nagiosxi

/bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php

su nagios
/bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
exit

[garethwithers]

Hi There please see below, I have put longer outputs into files and attached them:

Code: Select all

[root@vapngoa101 ~]# chage -l nagios
Last password change                                    : Mar 02, 2017
Password expires                                        : Mar 30, 2017
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 28
Number of days of warning before password expires       : 7

Code: Select all

[root@vapngoa101 ~]# cat /etc/passwd | grep nagios
nagios:x:54322:54322::/home/nagios:/bin/bash

Code: Select all

[root@vapngoa101 ~]# vi /etc/cron.d/nagiosxi

# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi

# Backup MySQL & PostgreSQL Databases
0   7 * * * root   /root/scripts/automysqlbackup
0   7 * * * root   /root/scripts/autopostgresqlbackup > /dev/null 2>&1

*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01  * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
*   * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1

Code: Select all

[root@vapngoa101 ~]# /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php

Code: Select all

[root@vapngoa101 ~]# su nagios
bash-4.1$ /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
exit

Thanks in advance for your help

Gareth

[mcapra]

The nagios user has an expired password:

Code: Select all

[root@vapngoa101 ~]# chage -l nagios
Last password change                                    : Mar 02, 2017
Password expires                                        : Mar 30, 2017

Can you run this command to remove the password expiration for the nagios user:

Code: Select all

chage -I -1 -m 0 -M 99999 -E -1 nagios

Then do a service crond restart, wait 4-5 minutes, then try to apply configuration again. Let us know if that doesn't fix things.

[garethwithers]

Hi thats fixed the issue thanks a lot for that.

Regards
Gareth

[cdienger]

Hi Gareth,

Glad to hear. Are we okay closing this thread?