Web UI stop working! ERR_EMPTY_RESPONSE

[IT_dog]

Hi to all
Recently our Nagios XI server (which working ok during last year) has been stopped shows many interface items like BBMap, Services grid, graphs etc.
Browser console says

GET http://nagiosxi.company.com/nagiosxi/ajaxhelper.php? <...> net::ERR_EMPTY_RESPONSE

It's cannot receive something from php backend.

Meanwhile httpd logs is like

Code: Select all

[Sun Jan 14 03:32:25.988406 2018] [:error] [pid 963] [client 192.168.2.52:52468] PHP Notice:  Use of undefined constant Servicegroup - assumed 'Servicegroup' in /usr/local/nagiosxi/html/includes/components/bbmap/bbmap.inc.php on line 194, referer: http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 14 03:32:25.995421 2018] [:error] [pid 674] [client 192.168.2.52:52467] PHP Notice:  Use of undefined constant Servicegroup - assumed 'Servicegroup' in /usr/local/nagiosxi/html/includes/components/bbmap/bbmap.inc.php on line 194, referer: http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 21 09:04:20.373729 2018] [core:notice] [pid 17907] AH00052: child pid 28459 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374851 2018] [core:notice] [pid 17907] AH00052: child pid 28396 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374915 2018] [core:notice] [pid 17907] AH00052: child pid 28424 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374928 2018] [core:notice] [pid 17907] AH00052: child pid 28425 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:22.376063 2018] [core:notice] [pid 17907] AH00052: child pid 28449 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:22.376125 2018] [core:notice] [pid 17907] AH00052: child pid 28923 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:23.378493 2018] [core:notice] [pid 17907] AH00052: child pid 28974 exit signal Segmentation fault (11)
[Sun Jan 21 08:59:17.335144 2018] [:error] [pid 17914] [client 94.251.35.8:43276] PHP Notice:  Array to string conversion in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 469, referer: 
http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 21 08:59:17.335222 2018] [:error] [pid 17914] [client 94.251.35.8:43276] PHP Notice:  Array to string conversion in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 606, referer: 
http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=

Everything else seems to be okay. What's going ? Please help.

I have already made
- db restore
- so many times httpd restarts
- full reinstall with db drop (after backup restoring the situation stay the same)

[dwhitfield]

- db restore
- so many times httpd restarts
- full reinstall with db drop (after backup restoring the situation stay the same)

Do you mean db repair when in the restore or are you just saying first you tried it without a drop. Can you attach a screenshot of what you see? Also, please take a look at https://assets.nagios.com/downloads/nag ... tabase.pdf

Please let us know if you have any trouble with the repair document.

[IT_dog]

I have tried both of db repair script and full db reinstall cycle (with deleting mariadb and dropping dbs from disk)

This is my problem:

(https://imgur.com/a/JCeDs full post)

[dwhitfield]

Most of the stuff in the log are notices, which can be ignored. Seeing the error UI, I think the issue may be fixed by changing some PHP settings.

In /etc/php.ini. please change the following to the listed numbers (unless the numbers are higher than these):

max_execution_time = 60
max_input_time = 120
memory_limit = 256M

Also, please add the following line:
max_input_vars = 10000

After making these changes you'll need to restart the httpd service:

Code: Select all

service httpd restart

If that does not resolve the issue, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.

You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh

That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.

After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.

If you get an error that PROFILE BUILD FAILED, please see https://support.nagios.com/kb/article.p ... ategory=44

[IT_dog]

Many thanks for the answer.
But while I have tried to find solution ... I lost the problem Just I don't made any changes and all widgets and dashboards began working as well themselves.

Anyway I've changed php.ini file and I will watch for this strange stuff and keep this topic updated.

Thank you

[IT_dog]

Well it is stop working again.

That is system info (stripped a little)

Code: Select all

Nagios XI - System Info
System:
Nagios XI Version : 5.4.11
nagiosxi 3.10.0-514.16.1.el7.x86_64 x86_64
CentOS Linux release 7.3.1611 (Core) 
Gnome is not installed

Apache Information
PHP Version: 5.4.16
Server Name: nagiosxi.xxx.com
Server Address: x.x.x.x
Server Port: 80

Nagios XI Data
License ends in: NTPRTO

nagios (pid 5715) is running...
NPCD running (pid 17931).
ndo2db (pid 17951) is running...
CPU Load 15: 0.39 
Total Hosts: 56 
Total Services: 941 
Function 'get_base_uri' returns: http://nagiosxi.xxx.com/nagiosxi/
Function 'get_base_url' returns: http://nagiosxi.xxx.com/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://nagiosxi.xxx.com/nagiosxi/includes/components/profile/profile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/

Ping Test localhost
Running:
/bin/ping -c 3 localhost 2>&1 
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.031 ms

--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.031/0.043/0.059/0.012 ms

Test wget To localhost
WGET From URL: http://localhost/nagiosxi/includes/components/ccm/ 
Running:
/usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/ 
--2018-01-24 10:36:34-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://localhost/nagiosxi/login.php?redirect=/nagiosxi/includes/components/ccm/index.php%3f&noauth=1 [following]
--2018-01-24 10:36:34-- http://localhost/nagiosxi/login.php?redirect=/nagiosxi/includes/components/ccm/index.php%3f&noauth=1
Reusing existing connection to [localhost]:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/usr/local/nagiosxi/tmp/ccm_index.tmp'

0K .......... .......... ..... 1.78M=0.01s

2018-01-24 10:36:34 (1.78 MB/s) - '/usr/local/nagiosxi/tmp/ccm_index.tmp' saved [26247]

[IT_dog]

I can't send you zip with system dump because it contain a lot of private info.
So any system information can be provided on demand ofc

[dwhitfield]

Is it possible that you can just delete the two tar.gz files in the profile and then zip up the actual logs? It's going to be very hard to resolve this without getting a complete picture of the server. Do you have any hardening on the system? Can you turn that off for troubleshooting purposes? It's well known that SELinux, modifying sudoers, and changing PAM settings for the system users XI uses causes problems. If we know what was changed we may be able to help you work around it, but ultimately, hardening will be up to you.

UPDATE: profile received and shared with techs

[IT_dog]

Sent some stripped info from exported profile.zip

[dwhitfield]

Please run through the following in order and let me know if you run into any issues.

NOTE: You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -f /usr/local/nagios/var/rw/nagios.cmd
# rm -f /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service mysqld start
# service ndo2db start
# service nagios start
# service httpd start
# service crond start

You definitely have kernel queue issues, so you'll need to increase the settings *LIKE* those in https://support.nagios.com/kb/article.php?id=139

I want to emphasize that the numbers in that document are conservative. If the numbers in the document do not resolve the issue, you may need to increase them more.