Applying Configuration issue when changing httpd under apach

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Applying Configuration issue when changing httpd under apach

Post by xlin125 »

We have a Nagios XI 2014R2.7 on RHEL 6.7. By default installation, the Apache web server httpd (parent, pid=1) runs by root. Due to the security concern, we changed the httpd configuration so that the httpd (parent, pid=1) runs by apache. The httpd started fine, and we were able to login to the Nagios XI web interface. However, when we made some changes (e.g., add/remove/update hosts/services), and click on "Apply Configuration", the Applying Configuration did not complete. It was on "Waiting for configuration verification................................................................................." forever. What was going on with this configuration verification? What was missing when changing the httpd to be run (parent, pid=1) by apache and how to resolve this issue? Any help to fix this issue will be greatly appreciated. Thanks in advance

httpd (pid=1) runs by apache
$ ps -ef|grep httpd
apache 947 1 0 Mar18 ? 00:00:03 /usr/sbin/httpd
apache 12414 947 0 01:33 ? 00:00:07 /usr/sbin/httpd
apache 13570 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 13578 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 13579 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 18874 18770 0 01:47 pts/2 00:00:00 grep httpd
apache 25114 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25115 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25116 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25117 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25118 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25119 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25120 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25122 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Applying Configuration issue when changing httpd under a

Post by lmiltchev »

Due to the security concern, we changed the httpd configuration so that the httpd (parent, pid=1) runs by apache.
Can you show us the exact modifications that you did to the system? We can try to recreate the issue in house. Knowing what you did would help.

Run the following commands and show the output:

Code: Select all

su nagios
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
Be sure to check out our Knowledgebase for helpful articles and solutions!
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Re: Applying Configuration issue when changing httpd under a

Post by xlin125 »

Imiltchev,

Thanks for the response. The following is the output of running /usr/local/nagiosxi/scripts/reconfigure_nagios.sh:
------------
[nagios@mtovis02 scripts]$ ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=396o2d' -O nagiosql.login--2016-03-21 13:39:32-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
NAGIOSQL LOGIN FAILED!
-----------

I noticed this failure on XI web interface under Admin->SYstem Config->System Profile as shown below but I was not sure where I can replace port 80 with port 8085, which is the new port that I changed to because httpd is run by apache so that the port has to be above 1024:
------------
Test wget To localhost
WGET From URL: http://localhost/nagiosxi/includes/components/ccm/
Running: /usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/
--2016-03-21 13:46:21-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
-----------

I installed the Nagios XI server using the XI 2014R2.7 package xi-latest.tar.gz. I made the following changes to the apache/httpd in order to have httpd run by apache on port 8085:

1. Stop apache
# service httpd stop
Stopping httpd: [ OK ]
2. Change the port number from 80 to 8085 since only root can use ports 1 – 1024
• Edit /etc/httpd/conf/httpd.conf
• Replace “Listen 80” with “Listen 8085”
# grep "Listen 8085" httpd.conf
Listen 8085
3. Change the own for /var/run/httpd directory from root to apache so that apache can create httpd.pid in this directory
# ls -ld /var/run/httpd
drwx--x--- 2 root apache 4096 Mar 20 21:43 /var/run/httpd
# chown apache /var/run/httpd
# ls -ld /var/run/httpd
drwx--x--- 2 apache apache 4096 Mar 20 21:43 /var/run/httpd
4. Change the ownship for /var/log/httpd and the files in this directory from root:root to apache:apache
# ls -ld /var/log/httpd
drwx------ 2 root root 4096 Mar 20 03:40 /var/log/httpd
# ls -l /var/log/httpd
total 146648
-rw-r--r-- 1 root root 530113 Mar 20 21:43 access_log
-rw-r--r-- 1 root root 20933495 Feb 28 03:26 access_log-20160228
-rw-r--r-- 1 root root 31571190 Mar 6 03:13 access_log-20160306
-rw-r--r-- 1 root root 59364744 Mar 13 03:33 access_log-20160313
-rw-r--r-- 1 root root 6217200 Mar 20 03:40 access_log-20160320
-rw-r--r-- 1 root root 3349 Mar 20 21:43 error_log
-rw-r--r-- 1 root root 45671 Feb 28 03:27 error_log-20160228
-rw-r--r-- 1 root root 499400 Mar 6 03:14 error_log-20160306
-rw-r--r-- 1 root root 28414976 Mar 13 03:34 error_log-20160313
-rw-r--r-- 1 root root 506 Mar 20 03:41 error_log-20160320
-rw-r--r-- 1 root root 0 Oct 10 07:41 ssl_access_log
-rw-r--r-- 1 root root 643137 Sep 29 16:11 ssl_access_log-20150930
-rw-r--r-- 1 root root 614220 Oct 9 16:11 ssl_access_log-20151010
-rw-r--r-- 1 root root 0 Oct 25 07:50 ssl_error_log
-rw-r--r-- 1 root root 27148 Sep 30 07:22 ssl_error_log-20151004
-rw-r--r-- 1 root root 27193 Oct 10 07:42 ssl_error_log-20151011
-rw-r--r-- 1 root root 229 Oct 11 07:09 ssl_error_log-20151018
-rw-r--r-- 1 root root 229 Oct 18 07:26 ssl_error_log-20151025
-rw-r--r-- 1 root root 0 Oct 10 07:41 ssl_request_log
-rw-r--r-- 1 root root 606257 Sep 29 16:11 ssl_request_log-20150930
-rw-r--r-- 1 root root 579176 Oct 9 16:11 ssl_request_log-20151010
# chown -R apache:apache /var/log/httpd
# ls -ld /var/log/httpd
drwx------ 2 apache apache 4096 Mar 20 03:40 /var/log/httpd
# ls -l /var/log/httpd
total 146648
-rw-r--r-- 1 apache apache 530113 Mar 20 21:43 access_log
-rw-r--r-- 1 apache apache 20933495 Feb 28 03:26 access_log-20160228
-rw-r--r-- 1 apache apache 31571190 Mar 6 03:13 access_log-20160306
-rw-r--r-- 1 apache apache 59364744 Mar 13 03:33 access_log-20160313
-rw-r--r-- 1 apache apache 6217200 Mar 20 03:40 access_log-20160320
-rw-r--r-- 1 apache apache 3349 Mar 20 21:43 error_log
-rw-r--r-- 1 apache apache 45671 Feb 28 03:27 error_log-20160228
-rw-r--r-- 1 apache apache 499400 Mar 6 03:14 error_log-20160306
-rw-r--r-- 1 apache apache 28414976 Mar 13 03:34 error_log-20160313
-rw-r--r-- 1 apache apache 506 Mar 20 03:41 error_log-20160320
-rw-r--r-- 1 apache apache 0 Oct 10 07:41 ssl_access_log
-rw-r--r-- 1 apache apache 643137 Sep 29 16:11 ssl_access_log-20150930
-rw-r--r-- 1 apache apache 614220 Oct 9 16:11 ssl_access_log-20151010
-rw-r--r-- 1 apache apache 0 Oct 25 07:50 ssl_error_log
-rw-r--r-- 1 apache apache 27148 Sep 30 07:22 ssl_error_log-20151004
-rw-r--r-- 1 apache apache 27193 Oct 10 07:42 ssl_error_log-20151011
-rw-r--r-- 1 apache apache 229 Oct 11 07:09 ssl_error_log-20151018
-rw-r--r-- 1 apache apache 229 Oct 18 07:26 ssl_error_log-20151025
-rw-r--r-- 1 apache apache 0 Oct 10 07:41 ssl_request_log
-rw-r--r-- 1 apache apache 606257 Sep 29 16:11 ssl_request_log-20150930
-rw-r--r-- 1 apache apache 579176 Oct 9 16:11 ssl_request_log-20151010
5. Change the lockfile directory defined in /etc/init.d/httpd
• Current lockfile path
# grep lockfile= /etc/init.d/httpd
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
• Edit /etc/init.d/httpd, and change the path to /var/log/httpd
# grep lockfile= /etc/init.d/httpd
## lockfile=${LOCKFILE-/var/lock/subsys/httpd}
lockfile=${LOCKFILE-/var/log/httpd}

6. Change ownship for /usr/sbin/httpd from root:root to apache:apache
# ls -l /usr/sbin/httpd
-rwxr-xr-x 1 root root 355232 Aug 15 2014 /usr/sbin/httpd
# chown apache:apache /usr/sbin/httpd
# ls -l /usr/sbin/httpd
-rwxr-xr-x 1 apache apache 355232 Aug 15 2014 /usr/sbin/httpd

7. Change the shell for apache so that we can sudo to apache
# grep apache /etc/passwd
apache:x:48:48:Apache:/var/www:/sbin/nologin
# usermod -s /bin/bash apache
# grep apache /etc/passwd
apache:x:48:48:Apache:/var/www:/bin/bash
8. Add 8085/tcp to iptables
9. Sudo to apache and start apache
# sudo su - apache
$ id
uid=48(apache) gid=48(apache) groups=48(apache),35954(nagios),35955(nagcmd)

$ service httpd start
Starting httpd: [ OK ]
$ ps -ef|grep httpd
apache 5313 1 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5781 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5782 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5783 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5784 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5785 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5786 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5787 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5788 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 6050 4233 0 22:22 pts/1 00:00:00 grep httpd
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Re: Applying Configuration issue when changing httpd under a

Post by xlin125 »

By the way, I disabled the SSL for Apache web server by renaming the file /etc/httpd/conf.d/ssl.conf to something else and restart httpd.
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Applying Configuration issue when changing httpd under a

Post by lmiltchev »

Try the following - open the "/usr/local/nagiosxi/html/config.inc.php" file in a text editor, and change this line:

Code: Select all

$cfg['port_number'] = false;
to this:

Code: Select all

$cfg['port_number'] = 8085;
Save, exit, and try to apply configuration again. Did it work this time?
Be sure to check out our Knowledgebase for helpful articles and solutions!
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Re: Applying Configuration issue when changing httpd under a

Post by xlin125 »

As per your instruction, I opened the "/usr/local/nagiosxi/html/config.inc.php" file in vi editor, and changed this line:
$cfg['port_number'] = false;

to

$cfg['port_number'] = 8085;

Then, I tried Apply Configuration again. I still got "Waiting for configuration verification .........................." on and on, and it did not end. So this issue still remains. Note that for verification purpose, I made the same changes to the Apache/httpd to use port 8085 and had the httpd run by apache on the 2nd Nagios XI server, and I got the same problem that Apply Configuration did not complete (in "Waiting for configuration verification" state"). So this problem is consistent. Please help look into it and let me know if you find a solution. Thanks!
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Applying Configuration issue when changing httpd under a

Post by lmiltchev »

I followed ALL of the steps you took very closely, and after the adding the port number to the config.inc.php I can apply configuration successfully. One more thing I did is to modify the program URL.
example01.PNG
example02.PNG
Can you run the "reconfigure_nagios.sh" script one more time, and show us the output?
You do not have the required permissions to view the files attached to this post.
Be sure to check out our Knowledgebase for helpful articles and solutions!
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Re: Applying Configuration issue when changing httpd under a

Post by xlin125 »

[nagios@mtovis02 scripts]$ ./reconfigure_nagios.sh
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=396o2d' -O nagiosql.login--2016-03-22 11:09:11-- http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8085... failed: Connection refused.
NAGIOSQL LOGIN FAILED!
------------

Did you get this connection refused failure on localhost when you ran ./reconfigure_nagios.sh? Before you added the port number 8085 to the config.inc.php, I guess you were unable to get Apply Configuration completed, correct? I did modify the Program URL as you did under Admin->Manage System Config.
User avatar
lmiltchev
Bugs find me
Posts: 13589
Joined: Mon May 23, 2011 12:15 pm

Re: Applying Configuration issue when changing httpd under a

Post by lmiltchev »

Before you added the port number 8085 to the config.inc.php, I guess you were unable to get Apply Configuration completed, correct?
That is correct. I was not able to apply configuration as I was getting the same error as you.

Try restarting httpd, and running reconfigure again.

Code: Select all

service httpd stop
su apache
service httpd start
exit
cd /usr/local/nagiosxi/scripts
./reconfigure_nagios.sh
Did it work this time (after restarting httpd)?

It's strange that it is working for me but not working for you, even though we should have the same setup...

Code: Select all

grep Listen /etc/httpd/conf/httpd.conf
# Listen: Allows you to bind Apache to specific IP addresses and/or
# Change this to Listen on specific IP addresses as shown below to
#Listen 12.34.56.78:80
#Listen 80
Listen 8085

Code: Select all

ps -ef | grep httpd
apache   25698     1  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25700 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25701 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25702 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25703 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25704 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25705 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25706 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
apache   25707 25698  0 11:16 ?        00:00:00 /usr/sbin/httpd
root     26558  1658  0 11:18 pts/0    00:00:00 grep httpd

Code: Select all

./reconfigure_nagios.sh
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O nagiosql.login--2016-03-22 11:27:26--  http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

    [ <=>                                                                        ] 35,366      --.-K/s   in 0.01s

2016-03-22 11:27:26 (2.80 MB/s) - “nagiosql.login” saved [35366]

LOGIN SUCCESSFUL!
IMPORTING CONFIG FILES...URL: http://localhost:8085/nagiosxi/includes/components/ccm/
Array
(
)
RESETTING PERMS
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O nagiosql.login--2016-03-22 11:27:27--  http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.login”

    [ <=>                                                                        ] 35,366      --.-K/s   in 0.01s

2016-03-22 11:27:27 (2.76 MB/s) - “nagiosql.login” saved [35366]

LOGIN SUCCESSFUL!
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE:
/usr/bin/wget --load-cookies=nagiosql.cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'cmd=apply&type=writeConfig' -O nagiosql.export.monitoring
--2016-03-22 11:27:27--  http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “nagiosql.export.monitoring”

    [ <=>                                                                        ] 14,951      86.3K/s   in 0.2s

2016-03-22 11:27:27 (86.3 KB/s) - “nagiosql.export.monitoring” saved [14951]

WRITE CONFIGS SUCCESSFUL!
OUTPUT:
Nagios Core 4.1.1
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-19-2015
License: GPL

Website: https://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 35 services.
        Checked 4 hosts.
        Checked 1 host groups.
        Checked 0 service groups.
        Checked 2 contacts.
        Checked 2 contact groups.
        Checked 124 commands.
        Checked 8 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 4 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 8 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check
RET: 0
Running configuration check...
Stopping nagios: done.
Starting nagios: done.
Be sure to check out our Knowledgebase for helpful articles and solutions!
xlin125
Posts: 172
Joined: Mon Jan 19, 2015 6:01 pm

Re: Applying Configuration issue when changing httpd under a

Post by xlin125 »

The ./reconfigure_nagios.sh still reported the same problem:

$ ./reconfigure_nagios.sh
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O nagiosql.login--2016-03-22 14:00:57-- http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8085... failed: Connection refused.
NAGIOSQL LOGIN FAILED!

Is this script used to verify/test the http connection to the localhost? What may possibly cause this failure? Thanks.
Locked