Nagios Support Forum

We have a Nagios XI 2014R2.7 on RHEL 6.7. By default installation, the Apache web server httpd (parent, pid=1) runs by root. Due to the security concern, we changed the httpd configuration so that the httpd (parent, pid=1) runs by apache. The httpd started fine, and we were able to login to the Nagios XI web interface. However, when we made some changes (e.g., add/remove/update hosts/services), and click on "Apply Configuration", the Applying Configuration did not complete. It was on "Waiting for configuration verification................................................................................." forever. What was going on with this configuration verification? What was missing when changing the httpd to be run (parent, pid=1) by apache and how to resolve this issue? Any help to fix this issue will be greatly appreciated. Thanks in advance

httpd (pid=1) runs by apache
$ ps -ef|grep httpd
apache 947 1 0 Mar18 ? 00:00:03 /usr/sbin/httpd
apache 12414 947 0 01:33 ? 00:00:07 /usr/sbin/httpd
apache 13570 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 13578 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 13579 947 0 01:36 ? 00:00:06 /usr/sbin/httpd
apache 18874 18770 0 01:47 pts/2 00:00:00 grep httpd
apache 25114 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25115 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25116 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25117 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25118 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25119 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd
apache 25120 947 0 Mar20 ? 00:00:06 /usr/sbin/httpd
apache 25122 947 0 Mar20 ? 00:00:07 /usr/sbin/httpd

Due to the security concern, we changed the httpd configuration so that the httpd (parent, pid=1) runs by apache.

Can you show us the exact modifications that you did to the system? We can try to recreate the issue in house. Knowing what you did would help.

Run the following commands and show the output:

Imiltchev,

Thanks for the response. The following is the output of running /usr/local/nagiosxi/scripts/reconfigure_nagios.sh:
------------
[nagios@mtovis02 scripts]$ ./reconfigure_nagios.sh
URL: http://localhost/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=396o2d' -O nagiosql.login--2016-03-21 13:39:32-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
NAGIOSQL LOGIN FAILED!
-----------

I noticed this failure on XI web interface under Admin->SYstem Config->System Profile as shown below but I was not sure where I can replace port 80 with port 8085, which is the new port that I changed to because httpd is run by apache so that the port has to be above 1024:
------------
Test wget To localhost
WGET From URL: http://localhost/nagiosxi/includes/components/ccm/
Running: /usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/
--2016-03-21 13:46:21-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... failed: Connection refused.
Connecting to localhost|127.0.0.1|:80... failed: Connection refused.
-----------

I installed the Nagios XI server using the XI 2014R2.7 package xi-latest.tar.gz. I made the following changes to the apache/httpd in order to have httpd run by apache on port 8085:

1. Stop apache
# service httpd stop
Stopping httpd: [ OK ]
2. Change the port number from 80 to 8085 since only root can use ports 1 – 1024
• Edit /etc/httpd/conf/httpd.conf
• Replace “Listen 80” with “Listen 8085”
# grep "Listen 8085" httpd.conf
Listen 8085
3. Change the own for /var/run/httpd directory from root to apache so that apache can create httpd.pid in this directory
# ls -ld /var/run/httpd
drwx--x--- 2 root apache 4096 Mar 20 21:43 /var/run/httpd
# chown apache /var/run/httpd
# ls -ld /var/run/httpd
drwx--x--- 2 apache apache 4096 Mar 20 21:43 /var/run/httpd
4. Change the ownship for /var/log/httpd and the files in this directory from root:root to apache:apache
# ls -ld /var/log/httpd
drwx------ 2 root root 4096 Mar 20 03:40 /var/log/httpd
# ls -l /var/log/httpd
total 146648
-rw-r--r-- 1 root root 530113 Mar 20 21:43 access_log
-rw-r--r-- 1 root root 20933495 Feb 28 03:26 access_log-20160228
-rw-r--r-- 1 root root 31571190 Mar 6 03:13 access_log-20160306
-rw-r--r-- 1 root root 59364744 Mar 13 03:33 access_log-20160313
-rw-r--r-- 1 root root 6217200 Mar 20 03:40 access_log-20160320
-rw-r--r-- 1 root root 3349 Mar 20 21:43 error_log
-rw-r--r-- 1 root root 45671 Feb 28 03:27 error_log-20160228
-rw-r--r-- 1 root root 499400 Mar 6 03:14 error_log-20160306
-rw-r--r-- 1 root root 28414976 Mar 13 03:34 error_log-20160313
-rw-r--r-- 1 root root 506 Mar 20 03:41 error_log-20160320
-rw-r--r-- 1 root root 0 Oct 10 07:41 ssl_access_log
-rw-r--r-- 1 root root 643137 Sep 29 16:11 ssl_access_log-20150930
-rw-r--r-- 1 root root 614220 Oct 9 16:11 ssl_access_log-20151010
-rw-r--r-- 1 root root 0 Oct 25 07:50 ssl_error_log
-rw-r--r-- 1 root root 27148 Sep 30 07:22 ssl_error_log-20151004
-rw-r--r-- 1 root root 27193 Oct 10 07:42 ssl_error_log-20151011
-rw-r--r-- 1 root root 229 Oct 11 07:09 ssl_error_log-20151018
-rw-r--r-- 1 root root 229 Oct 18 07:26 ssl_error_log-20151025
-rw-r--r-- 1 root root 0 Oct 10 07:41 ssl_request_log
-rw-r--r-- 1 root root 606257 Sep 29 16:11 ssl_request_log-20150930
-rw-r--r-- 1 root root 579176 Oct 9 16:11 ssl_request_log-20151010
# chown -R apache:apache /var/log/httpd
# ls -ld /var/log/httpd
drwx------ 2 apache apache 4096 Mar 20 03:40 /var/log/httpd
# ls -l /var/log/httpd
total 146648
-rw-r--r-- 1 apache apache 530113 Mar 20 21:43 access_log
-rw-r--r-- 1 apache apache 20933495 Feb 28 03:26 access_log-20160228
-rw-r--r-- 1 apache apache 31571190 Mar 6 03:13 access_log-20160306
-rw-r--r-- 1 apache apache 59364744 Mar 13 03:33 access_log-20160313
-rw-r--r-- 1 apache apache 6217200 Mar 20 03:40 access_log-20160320
-rw-r--r-- 1 apache apache 3349 Mar 20 21:43 error_log
-rw-r--r-- 1 apache apache 45671 Feb 28 03:27 error_log-20160228
-rw-r--r-- 1 apache apache 499400 Mar 6 03:14 error_log-20160306
-rw-r--r-- 1 apache apache 28414976 Mar 13 03:34 error_log-20160313
-rw-r--r-- 1 apache apache 506 Mar 20 03:41 error_log-20160320
-rw-r--r-- 1 apache apache 0 Oct 10 07:41 ssl_access_log
-rw-r--r-- 1 apache apache 643137 Sep 29 16:11 ssl_access_log-20150930
-rw-r--r-- 1 apache apache 614220 Oct 9 16:11 ssl_access_log-20151010
-rw-r--r-- 1 apache apache 0 Oct 25 07:50 ssl_error_log
-rw-r--r-- 1 apache apache 27148 Sep 30 07:22 ssl_error_log-20151004
-rw-r--r-- 1 apache apache 27193 Oct 10 07:42 ssl_error_log-20151011
-rw-r--r-- 1 apache apache 229 Oct 11 07:09 ssl_error_log-20151018
-rw-r--r-- 1 apache apache 229 Oct 18 07:26 ssl_error_log-20151025
-rw-r--r-- 1 apache apache 0 Oct 10 07:41 ssl_request_log
-rw-r--r-- 1 apache apache 606257 Sep 29 16:11 ssl_request_log-20150930
-rw-r--r-- 1 apache apache 579176 Oct 9 16:11 ssl_request_log-20151010
5. Change the lockfile directory defined in /etc/init.d/httpd
• Current lockfile path
# grep lockfile= /etc/init.d/httpd
lockfile=${LOCKFILE-/var/lock/subsys/httpd}
• Edit /etc/init.d/httpd, and change the path to /var/log/httpd
# grep lockfile= /etc/init.d/httpd
## lockfile=${LOCKFILE-/var/lock/subsys/httpd}
lockfile=${LOCKFILE-/var/log/httpd}

6. Change ownship for /usr/sbin/httpd from root:root to apache:apache
# ls -l /usr/sbin/httpd
-rwxr-xr-x 1 root root 355232 Aug 15 2014 /usr/sbin/httpd
# chown apache:apache /usr/sbin/httpd
# ls -l /usr/sbin/httpd
-rwxr-xr-x 1 apache apache 355232 Aug 15 2014 /usr/sbin/httpd

7. Change the shell for apache so that we can sudo to apache
# grep apache /etc/passwd
apache48:48:Apache:/var/www:/sbin/nologin
# usermod -s /bin/bash apache
# grep apache /etc/passwd
apache48:48:Apache:/var/www:/bin/bash
8. Add 8085/tcp to iptables
9. Sudo to apache and start apache
# sudo su - apache
$ id
uid=48(apache) gid=48(apache) groups=48(apache),35954(nagios),35955(nagcmd)

$ service httpd start
Starting httpd: [ OK ]
$ ps -ef|grep httpd
apache 5313 1 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5781 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5782 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5783 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5784 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5785 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5786 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5787 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 5788 5313 0 22:21 ? 00:00:00 /usr/sbin/httpd
apache 6050 4233 0 22:22 pts/1 00:00:00 grep httpd

By the way, I disabled the SSL for Apache web server by renaming the file /etc/httpd/conf.d/ssl.conf to something else and restart httpd.

Try the following - open the "/usr/local/nagiosxi/html/config.inc.php" file in a text editor, and change this line: to this: Save, exit, and try to apply configuration again. Did it work this time?

As per your instruction, I opened the "/usr/local/nagiosxi/html/config.inc.php" file in vi editor, and changed this line:
$cfg['port_number'] = false;

to

$cfg['port_number'] = 8085;

Then, I tried Apply Configuration again. I still got "Waiting for configuration verification .........................." on and on, and it did not end. So this issue still remains. Note that for verification purpose, I made the same changes to the Apache/httpd to use port 8085 and had the httpd run by apache on the 2nd Nagios XI server, and I got the same problem that Apply Configuration did not complete (in "Waiting for configuration verification" state"). So this problem is consistent. Please help look into it and let me know if you find a solution. Thanks!

I followed ALL of the steps you took very closely, and after the adding the port number to the config.inc.php I can apply configuration successfully. One more thing I did is to modify the program URL.
Can you run the "reconfigure_nagios.sh" script one more time, and show us the output?

[nagios@mtovis02 scripts]$ ./reconfigure_nagios.sh
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=396o2d' -O nagiosql.login--2016-03-22 11:09:11-- http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8085... failed: Connection refused.
NAGIOSQL LOGIN FAILED!
------------

Did you get this connection refused failure on localhost when you ran ./reconfigure_nagios.sh? Before you added the port number 8085 to the config.inc.php, I guess you were unable to get Apply Configuration completed, correct? I did modify the Program URL as you did under Admin->Manage System Config.

Before you added the port number 8085 to the config.inc.php, I guess you were unable to get Apply Configuration completed, correct?

That is correct. I was not able to apply configuration as I was getting the same error as you.

Try restarting httpd, and running reconfigure again.
Did it work this time (after restarting httpd)?

It's strange that it is working for me but not working for you, even though we should have the same setup...

The ./reconfigure_nagios.sh still reported the same problem:

$ ./reconfigure_nagios.sh
URL: http://localhost:8085/nagiosxi/includes/components/ccm/
CMDLINE
/usr/bin/wget --save-cookies nagiosql.cookies --keep-session-cookies http://localhost:8085/nagiosxi/includes/components/ccm/ --no-check-certificate --post-data 'submit=Login&hidelog=true&loginSubmitted=true&username=nagiosxi&password=n@gweb' -O nagiosql.login--2016-03-22 14:00:57-- http://localhost:8085/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8085... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8085... failed: Connection refused.
NAGIOSQL LOGIN FAILED!

Is this script used to verify/test the http connection to the localhost? What may possibly cause this failure? Thanks.