Nagios Support Forum

Hi all,
I'm new here and registered because I'm running out of ideas for the following problem:
I'm running a Nagios 3.5.1, installed via yum from an EPEL-repo on a RedHat 7.1-system.
I moved the log-files and configuration to a different location (separate disk). I enabled LDAP-Auth for the GUI in the /etc/httpd/conf.d/nagios.conf.
Everything is working pretty well except of one thing:
From time to time (~ every 30 refreshes) or when I click somewhere on the dashboard, I'm getting the "Error 500" and I can still see the menu on the right side, but no hosts or status-infos.
In the httpd-error-log I can see this at the same time e.g.:
[Thu Jan 14 12:00:16.239471 2016] [cgi:error] [pid 5026] [client 192.168.2.100:61945] End of script output before headers: status.cgi, referer: http://nagios001/nagios/cgi-bin//status ... ustypes=15
or:
[Thu Jan 14 11:09:06.246553 2016] [cgi:error] [pid 27117] [client 192.168.2.100:61203] End of script output before headers: status.cgi, referer: http://nagios001/nagios/side.php

From different posts, I could see, that it might has something to do with the file-permissions or incorrect configuration, but I check eversything I could find.

The strange thing is, that this error occurs only sometimes and not permanently. So my question is:
Could it be possible, that this is due to a high load of the nagios during executing checks or any file-locks or does anyone have another idea?

Thank you all in advance

BR

I moved the log-files and configuration to a different location (separate disk)

What files did you move to a different location?

Was this happening before, or after you integrated LDAP? Can you post your apache configuration files?

I moved all configuration-files for:
Services-definition, Servicegroups-definition, Hosts-definition, Hostgroups-definition, Command-definition, Contact-definition, Checktime-definition
And I moved the nagios.log and the status.dat files.

Which config-files exactly?

This happend also before the LDAP-integration.

What are the permissions on the status.cgi file?

Do you have selinux enabled? getenforce

How about suExec? We have seen that cause issues in the past: https://support.nagios.com/forum/viewto ... 20#p142073

https://httpd.apache.org/docs/2.2/suexec.html#enable

Permissions on status.cgi:
ls -lah /usr/lib64/nagios/cgi-bin/status.cgi
-rwxr-xr-x. 1 root root 300K Feb 7 2014 /usr/lib64/nagios/cgi-bin/status.cgi

selinux is disabled.

The phpinfo() is giving me the following result:
Server API: Apache 2.0 Handler
So the server should not run in suexec-mode right?

I believe the permissions need to be nagios:nagios, can you also post the output of the rest of the folder?

nagios001:/usr/lib64/nagios/cgi-bin# ls -lah
total 4.0M
drwxrwxr-x. 2 root root 4.0K Jan 14 09:39 .
drwxr-xr-x. 4 root root 34 Nov 5 11:34 ..
-rwxrwxr-x. 1 root root 288K Feb 7 2014 avail.cgi
-rwxrwxr-x. 1 root root 286K Feb 7 2014 cmd.cgi
-rwxrwxr-x. 1 root root 256K Feb 7 2014 config.cgi
-rwxrwxr-x. 1 root root 300K Feb 7 2014 extinfo.cgi
-rwxrwxr-x. 1 root root 251K Feb 7 2014 histogram.cgi
-rwxrwxr-x. 1 root root 227K Feb 7 2014 history.cgi
-rwxrwxr-x. 1 root root 227K Feb 7 2014 notifications.cgi
-rwxrwxr-x. 1 root root 219K Feb 7 2014 outages.cgi
-rwxrwxr-x. 1 root root 223K Feb 7 2014 showlog.cgi
-rwxr-xr-x. 1 nagios nagios 300K Feb 7 2014 status.cgi
-rwxrwxr-x. 1 root root 248K Feb 7 2014 statusmap.cgi
-rwxrwxr-x. 1 root root 240K Feb 7 2014 statuswml.cgi
-rwxrwxr-x. 1 root root 228K Feb 7 2014 statuswrl.cgi
-rwxrwxr-x. 1 root root 248K Feb 7 2014 summary.cgi
-rwxrwxr-x. 1 root root 240K Feb 7 2014 tac.cgi
-rwxrwxr-x. 1 root root 256K Feb 7 2014 trends.cgi

Changing the permissions didn't solve the problem.
As the problem is not occuring on every GUI-update, I really don't think it's a problem with the permissions or wrong config. Could it be due to high load?

Yes, this could be the culprit, especially as this is happening at random.

What is the result of top|head -5? Also, how many CPUs do you have allocated to the machine?

top|head -5
top - 07:21:43 up 35 days, 21:02, 4 users, load average: 8.12, 5.53, 4.29
Tasks: 173 total, 2 running, 168 sleeping, 3 stopped, 0 zombie
%Cpu(s): 5.7 us, 1.5 sy, 0.0 ni, 92.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8011552 total, 1974816 free, 775024 used, 5261712 buff/cache
KiB Swap: 1257468 total, 1256288 free, 1180 used. 6589452 avail Mem

The VM has 4 CPU-Cores. Should I consider upgrading the CPUs or the RAM or both? There is also an alert for the localhost-load on this server.

Short update:
I've upgraded the RAM to 12GB. Still have Four CPU-cores:

top - 13:50:47 up 1:11, 2 users, load average: 0.02, 0.06, 0.11
Tasks: 184 total, 3 running, 181 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.1 us, 1.3 sy, 0.0 ni, 92.5 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 12140304 total, 11242764 free, 434120 used, 463420 buff/cache
KiB Swap: 1257468 total, 1257468 free, 0 used. 11446752 avail Mem