CGI and Internal Server Errors

[scrusi]

Hey all,

I've been banging my ahead against a fresh install of Nagios Core for most of today and simply can't find a solution. There are lots of threads on this issue, but I can't seem to find a solution that works.

Situation:
Nagios 4.0.8 installed from source on Ubuntu 14.04 with Plesk
Nagios process seems to run fine.
Config check returns no errors. (Tried both default config files and my old ones.)
Webinterface loads, but returns HTTP 500 on all relevant pages. Home shows "Unable to get process status".
Apache error log shows the following:

Code: Select all

 [cgi:error] [pid 31659] [client myip:63760] End of script output before headers: status.cgi

Corresponding access log:

Code: Select all

myip - nagiosadmin [10/Jun/2015:20:02:40 +0200] "GET /nagios/cgi-bin/status.cgi?host=all HTTP/1.1" 500 764 "http://mydomain/nagios/side.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"

Running the same cgi from shell produces results without errors. (Tried running as root, nagios, and www-data [apache user])
SElinux does not seem to be installed.

ls -la /usr/local/nagios/share/ :

Code: Select all

total 100
drwxrwxr-x 11 nagios nagios 4096 Jun 10 19:02 .
drwxr-xr-x  9 root   root   4096 Jun 10 17:15 ..
-rw-rw-r--  1 nagios nagios  576 Jun 10 19:02 config.inc.php
drwxrwxr-x  2 nagios nagios 4096 Jun 10 19:02 contexthelp
drwxrwxr-x  3 nagios nagios 4096 Jun 10 17:03 docs
drwxrwxr-x  3 nagios nagios 4096 Jun 10 19:02 images
drwxrwxr-x  3 nagios nagios 4096 Jun 10 19:02 includes
-rw-rw-r--  1 nagios nagios 1628 Jun 10 19:02 index.php
drwxrwxr-x  2 nagios nagios 4096 Jun 10 19:02 js
-rw-rw-r--  1 nagios nagios 1176 Jun 10 19:02 jsonquery.html
drwxr-xr-x  4 nagios nagios 4096 Jun 10 17:15 locale
-rw-rw-r--  1 nagios nagios 8697 Jun 10 19:02 main.php
drwxrwxr-x  2 nagios nagios 4096 Jun 10 17:03 media
-rw-rw-r--  1 nagios nagios   26 Jun 10 19:02 robots.txt
-rw-rw-r--  1 nagios nagios  656 Jun 10 19:02 rss-corebanner.php
-rw-rw-r--  1 nagios nagios 1064 Jun 10 19:02 rss-corefeed.html
-rw-rw-r--  1 nagios nagios  895 Jun 10 19:02 rss-corefeed.php
-rw-rw-r--  1 nagios nagios  540 Jun 10 19:02 rss-newsfeed.html
-rw-rw-r--  1 nagios nagios 1064 Jun 10 19:02 rss-newsfeed.php
-rw-rw-r--  1 nagios nagios 6009 Jun 10 19:02 side.php
drwxrwxr-x  2 nagios nagios 4096 Jun 10 17:03 ssi
drwxrwxr-x  2 nagios nagios 4096 Jun 10 19:02 stylesheets

ls -la /usr/local/nagios/sbin/:

Code: Select all

total 4364
drwxrwxr-x 2 nagios nagios   4096 Jun 10 19:02 .
drwxr-xr-x 9 root   root     4096 Jun 10 17:15 ..
-rwxrwxr-x 1 nagios nagios 320576 Jun 10 19:02 archivejson.cgi
-rwxrwxr-x 1 nagios nagios 301880 Jun 10 19:02 avail.cgi
-rwxrwxr-x 1 nagios nagios 300296 Jun 10 19:02 cmd.cgi
-rwxrwxr-x 1 nagios nagios 273136 Jun 10 19:02 config.cgi
-rwxrwxr-x 1 nagios nagios 314136 Jun 10 19:02 extinfo.cgi
-rwxrwxr-x 1 nagios nagios 244504 Jun 10 19:02 history.cgi
-rwxrwxr-x 1 nagios nagios 240400 Jun 10 19:02 notifications.cgi
-rwxrwxr-x 1 nagios nagios 318272 Jun 10 19:02 objectjson.cgi
-rwxrwxr-x 1 nagios nagios 236264 Jun 10 19:02 outages.cgi
-rwxrwxr-x 1 nagios nagios 240376 Jun 10 19:02 showlog.cgi
-rwxrwxr-x 1 nagios nagios 314160 Jun 10 19:02 status.cgi
-rwxrwxr-x 1 nagios nagios 316416 Jun 10 19:02 statusjson.cgi
-rwxrwxr-x 1 nagios nagios 256816 Jun 10 19:02 statuswml.cgi
-rwxrwxr-x 1 nagios nagios 240400 Jun 10 19:02 statuswrl.cgi
-rwxrwxr-x 1 nagios nagios 265024 Jun 10 19:02 summary.cgi
-rwxrwxr-x 1 nagios nagios 252736 Jun 10 19:02 tac.cgi

cat /etc/group | grep nag:

Code: Select all

nagios:x:10001:
nagcmd:x:10002:nagios,www-data

grep authori cgi.cfg | grep -v '#':

Code: Select all

authorized_for_system_information=nagiosadmin
authorized_for_configuration_information=nagiosadmin
authorized_for_system_commands=nagiosadmin
authorized_for_all_services=nagiosadmin
authorized_for_all_hosts=nagiosadmin
authorized_for_all_service_commands=nagiosadmin
authorized_for_all_host_commands=nagiosadmin

ls -la /usr/local/nagios/var/rw:

Code: Select all

total 8
drwxrwsr-x 2 nagios nagcmd 4096 Jun 10 19:06 .
drwxrwxr-x 5 nagios nagios 4096 Jun 10 20:22 ..
prwxrwx--- 1 nagios nagcmd    0 Jun 10 19:06 nagios.cmd
srwxrwx--- 1 nagios nagcmd    0 Jun 10 19:06 nagios.qh

Running a pretty basic apache conf (through plesk):

Code: Select all

ScriptAlias /nagios/cgi-bin "/usr/local/nagios/sbin"

<Directory "/usr/local/nagios/sbin">
	#  SSLRequireSSL
	Options ExecCGI
	AllowOverride None
	Order allow,deny
	Allow from all
	#  Order deny,allow
	#  Deny from all
	#  Allow from 127.0.0.1
	AuthName "Nagios Access"
	AuthType Basic
	AuthUserFile /usr/local/nagios/etc/htpasswd.users
	Require valid-user
</Directory>

Alias /nagios "/usr/local/nagios/share"

<Directory "/usr/local/nagios/share">
	#  SSLRequireSSL
	Options None
	AllowOverride None
	Order allow,deny
	Allow from all
	#  Order deny,allow
	#  Deny from all
	#  Allow from 127.0.0.1
	AuthName "Nagios Access"
	AuthType Basic
	AuthUserFile /usr/local/nagios/etc/htpasswd.users
	Require valid-user
</Directory>

I'm almost certain this is a permissions/security issue but I just can't figure out where the problem is. This is not my first time setting up Nagios, but the first time on an ubuntu/plesk machine.

Any ideas would be very welcome.

[jolson]

Here is a full list of known good commands for getting Nagios working on 14.04:

Code: Select all

apt-get --ignore-missing install wget build-essential apache2 php5-gd libgd2-xpm-dev libapache2-mod-php5 unzip m4 automake libtool gettext apache2-utils
cd /tmp
wget https://github.com/NagiosEnterprises/nagioscore/archive/master.tar.gz -O nagioscore.tar.gz
wget https://github.com/nagios-plugins/nagios-plugins/archive/master.tar.gz -O nagiosplugins.tar.gz
useradd nagios
groupadd nagcmd
usermod -a -G nagcmd nagios
usermod -a -G nagcmd www-data
tar xzf nagioscore.tar.gz
tar xzf nagiosplugins.tar.gz
cd nagioscore-master
./configure --with-command-group=nagcmd --with-nagios-group=nagios --with-mail=/usr/bin/sendmail
make all
make install
make install-init
make install-config
make install-commandmode
/usr/bin/install -c -m 644 sample-config/httpd.conf /etc/apache2/sites-enabled/nagios.conf
cp -R contrib/eventhandlers /usr/local/nagios/libexec
chown -R nagios:nagios /usr/local/nagios/libexec/eventhandlers
a2enmod rewrite
a2enmod cgi
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
/etc/init.d/nagios start 
htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin
/etc/init.d/apache2 restart
cd /tmp/nagios-plugins-master/
./tools/setup
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install
ln -s /etc/init.d/apache2 /etc/rcS.d/S99apache2
ln -s /etc/init.d/nagios /etc/rcS.d/S99nagios

Please compare the above with your own history and double-check to make sure you didn't miss a step. If all else fails, you could re-compile and re-install per the above instructions.

[ssax]

In addition to jolson's post, what is the output of this command:

Code: Select all

grep "nag\|apache\|psaserv" /etc/group

[scrusi]

Hey,

thanks for the quick replies! I'll double check the history tomorrow, don't have access to it from home. Process seems very similar to what I did, with some minor differences (but not new steps) as far as I can remember. If it turns out to be significantly different, I'll try a clean install with your process tomorrow.

Here's the requested output from grep "nag\|apache\|psaserv" /etc/group :

Code: Select all

lock-manager:x:999:psaadm
psaserv:x:1002:psaadm,psaftp,www-data
nagios:x:10001:
nagcmd:x:10002:nagios,www-data

Thanks!

[jolson]

Sounds good - let us know what you find out scrusi. Thanks!

[scrusi]

Morning,
checked my history. Differences were in the source used, the tools setup, and the way the configuration is incorporated into apache. I have removed nagios and followed your instructions step-by step. Unfortunately, the problem is identical.
Anything else I could check?

Greetings,
scrusi

[jolson]

Is there anything else that you added to the setup - you mentioned Plesk previously. Is that still in the mix of things?

It would be good to check out the http error log:

Code: Select all

tail -n20 /var/log/httpd/error_log

[scrusi]

The only change I made in plesk was loading the httpd.conf. I removed that part when I removed nagios and used your suggestion of adding it straight to sites-enabled instead. I could still see plesk mucking things up, but not due to the previous install of nagios.

Here's the error log (really just more of what I posted above):

Code: Select all

[Thu Jun 11 19:26:48.787365 2015] [cgi:error] [pid 29986] [client myIP:63392] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:26:49.310620 2015] [cgi:error] [pid 29941] [client myIP:63393] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:26:50.221202 2015] [cgi:error] [pid 30107] [client myIP:63394] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:26:52.405617 2015] [cgi:error] [pid 30107] [client myIP:63400] End of script output before headers: statusjson.cgi, referer: http://mydomain/nagios/map.html?host=all
[Thu Jun 11 19:26:52.471925 2015] [cgi:error] [pid 29941] [client myIP:63398] End of script output before headers: statusjson.cgi, referer: http://mydomain/nagios/map.html?host=all
[Thu Jun 11 19:26:52.476952 2015] [cgi:error] [pid 29986] [client myIP:63399] End of script output before headers: objectjson.cgi, referer: http://mydomain/nagios/map.html?host=all
[Thu Jun 11 19:26:52.518726 2015] [cgi:error] [pid 30209] [client myIP:63401] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:00.034527 2015] [cgi:error] [pid 29941] [client myIP:63405] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:00.819372 2015] [cgi:error] [pid 29986] [client myIP:63406] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:01.725991 2015] [cgi:error] [pid 30209] [client myIP:63408] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:02.168339 2015] [cgi:error] [pid 29338] [client myIP:63409] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:13.174102 2015] [cgi:error] [pid 30105] [client myIP:63413] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:13.621646 2015] [cgi:error] [pid 29986] [client myIP:63415] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:14.045856 2015] [cgi:error] [pid 30209] [client myIP:63416] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:14.588604 2015] [cgi:error] [pid 30107] [client myIP:63417] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:15.109571 2015] [cgi:error] [pid 30105] [client myIP:63419] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:15.621164 2015] [cgi:error] [pid 29338] [client myIP:63420] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:16.081383 2015] [cgi:error] [pid 29986] [client myIP:63421] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:16.726443 2015] [cgi:error] [pid 30209] [client myIP:63422] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php
[Thu Jun 11 19:27:17.070150 2015] [cgi:error] [pid 30107] [client myIP:63423] End of script output before headers: status.cgi, referer: http://mydomain/nagios/side.php

[ssax]

Try editing your /etc/group and change:

Code: Select all

nagios:x:10001:

To:

Code: Select all

nagios:x:10001:nagios,www-data

Then restart plesk/httpd/nagios and try that.

[scrusi]

grep "nag\|apache\|psaserv" /etc/group now:

Code: Select all

lock-manager:x:999:psaadm
psaserv:x:1002:psaadm,psaftp,www-data
nagios:x:10001:nagios,www-data
nagcmd:x:10002:nagios,www-data

Code: Select all

~# /etc/init.d/apache2 restart
 * Restarting web server apache2                                                                                                       
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/plesk.conf.d/server.conf:6
                                                                                                                                [ OK ]
~# /etc/init.d/psa restart
 * Restarting Plesk engine pool manager sw-engine-fpm                                                                           [ OK ]
~# /etc/init.d/nagios restart
Running configuration check...
Stopping nagios: done.
Starting nagios: done.

Unfortunately no change.

Thanks for the help so far!