Page 1 of 1
Apply configuration not completing seems to be hanging
Posted: Wed Apr 05, 2017 8:02 am
by garethwithers
Afternoon
We have recently noticed that our apply configuration is not working could someone point me in the direction to troubleshoot this other than here:
https://support.nagios.com/kb/article.p ... ategory=44
Have tried all of these with no joy
Kind Regards
Gareth
Re: Apply configuration not completing seems to be hanging
Posted: Wed Apr 05, 2017 2:25 pm
by mcapra
Can you share what the tail that KB mentions displays when you run an "Apply Configuration"?
Code: Select all
tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Could you also share a system profile? Either via PM or by posting it as an attachment. From the Nagios XI GUI, you can gather a profile via Admin -> System Profile -> Download Profile.
Re: Apply configuration not completing seems to be hanging
Posted: Thu Apr 06, 2017 4:26 am
by garethwithers
I have included the profile in the ticket and below is the output from the command, there is nothing since a backup was completed yesterday:
Code: Select all
[root@vapngoa101 ~]# tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Starting nagios: done.
Backing up Core Config Manager (NagiosQL)...
tar: Removing leading `/' from member names
tar: Removing leading `/' from member names
Backing up Nagios Core...
tar: Removing leading `/' from member names
tar: /usr/local/nagios/var/rw/nagios.qh: socket ignored
tar: /usr/local/nagios/var/ndo.sock: socket ignored
Backing up Nagios XI...
tar: Removing leading `/' from member names
So the apply configuration is not writing to that file
Gareth
Re: Apply configuration not completing seems to be hanging
Posted: Thu Apr 06, 2017 12:25 pm
by mcapra
Can you share the outputs of:
Code: Select all
service crond status
service crond restart
#give the system 1-2 minutes to settle
ps aux | grep php
Can you also send over the contents of your apache logs? They're typically located in /var/log/httpd.
Re: Apply configuration not completing seems to be hanging
Posted: Fri Apr 07, 2017 2:41 am
by garethwithers
Hi there
Outputs below:
Code: Select all
[root@vapngoa101 ~]# service crond status
crond (pid 1899) is running...
[root@vapngoa101 ~]# service crond restart
Stopping crond: [ OK ]
Starting crond: [ OK ]
[root@vapngoa101 ~]# ps aux | grep php
root 120600 0.0 0.0 103316 832 pts/0 S+ 08:37 0:00 grep php
Logs attached
Re: Apply configuration not completing seems to be hanging
Posted: Fri Apr 07, 2017 10:09 am
by mcapra
The issue is that there are PHP scripts that run as cron jobs under the nagios user that are responsible for handing the "Apply Configuration" job (among other things), and they are not running for some reason.
Can you share the outputs of these commands:
Code: Select all
cat /etc/php.ini
php -r 'phpinfo();'
chage -l nagios
cat /etc/passwd | grep nagios
vi /etc/cron.d/nagiosxi
/bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
su nagios
/bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
exit
Re: Apply configuration not completing seems to be hanging
Posted: Mon Apr 10, 2017 7:49 am
by garethwithers
Hi There please see below, I have put longer outputs into files and attached them:
Code: Select all
[root@vapngoa101 ~]# chage -l nagios
Last password change : Mar 02, 2017
Password expires : Mar 30, 2017
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 28
Number of days of warning before password expires : 7
Code: Select all
[root@vapngoa101 ~]# cat /etc/passwd | grep nagios
nagios:x:54322:54322::/home/nagios:/bin/bash
Code: Select all
[root@vapngoa101 ~]# vi /etc/cron.d/nagiosxi
# /etc/cron.d/nagiosxi: crontab fragment for nagiosxi
# Backup MySQL & PostgreSQL Databases
0 7 * * * root /root/scripts/automysqlbackup
0 7 * * * root /root/scripts/autopostgresqlbackup > /dev/null 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/sysstat.php > /usr/local/nagiosxi/var/sysstat.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php > /usr/local/nagiosxi/var/cmdsubsys.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/eventman.php > /usr/local/nagiosxi/var/eventman.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/event_handler.php > /usr/local/nagiosxi/var/event_handler.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/feedproc.php > /usr/local/nagiosxi/var/feedproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/perfdataproc.php > /usr/local/nagiosxi/var/perfdataproc.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/nom.php > /usr/local/nagiosxi/var/nom.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/reportengine.php > /usr/local/nagiosxi/var/reportengine.log 2>&1
*/5 * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/dbmaint.php > /usr/local/nagiosxi/var/dbmaint.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/cleaner.php > /usr/local/nagiosxi/var/cleaner.log 2>&1
01 * * * * nagios /usr/local/nagiosxi/cron/recurringdowntime.pl > /usr/local/nagiosxi/var/recurringdowntime.log 2>&1
* * * * * nagios /usr/bin/php -q /usr/local/nagiosxi/cron/deadpool.php > /usr/local/nagiosxi/var/deadpool.log 2>&1
Code: Select all
[root@vapngoa101 ~]# /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
Code: Select all
[root@vapngoa101 ~]# su nagios
bash-4.1$ /bin/sh -c /usr/bin/php -q /usr/local/nagiosxi/cron/cmdsubsys.php
exit
Thanks in advance for your help
Gareth
Re: Apply configuration not completing seems to be hanging
Posted: Mon Apr 10, 2017 2:17 pm
by mcapra
The nagios user has an expired password:
Code: Select all
[root@vapngoa101 ~]# chage -l nagios
Last password change : Mar 02, 2017
Password expires : Mar 30, 2017
Can you run this command to remove the password expiration for the nagios user:
Code: Select all
chage -I -1 -m 0 -M 99999 -E -1 nagios
Then do a
service crond restart, wait 4-5 minutes, then try to apply configuration again. Let us know if that doesn't fix things.
Re: Apply configuration not completing seems to be hanging
Posted: Tue Apr 11, 2017 2:44 am
by garethwithers
Hi thats fixed the issue thanks a lot for that.
Regards
Gareth
Re: Apply configuration not completing seems to be hanging
Posted: Tue Apr 11, 2017 10:38 am
by cdienger
Hi Gareth,
Glad to hear. Are we okay closing this thread?