Page 1 of 3
Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Sun Jan 21, 2018 9:11 am
by IT_dog
Hi to all
Recently our Nagios XI server (which working ok during last year) has been stopped shows many interface items like BBMap, Services grid, graphs etc.
Browser console says
GET
http://nagiosxi.company.com/nagiosxi/ajaxhelper.php? <...> net::ERR_EMPTY_RESPONSE
It's cannot receive something from php backend.
Meanwhile httpd logs is like
Code: Select all
[Sun Jan 14 03:32:25.988406 2018] [:error] [pid 963] [client 192.168.2.52:52468] PHP Notice: Use of undefined constant Servicegroup - assumed 'Servicegroup' in /usr/local/nagiosxi/html/includes/components/bbmap/bbmap.inc.php on line 194, referer: http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 14 03:32:25.995421 2018] [:error] [pid 674] [client 192.168.2.52:52467] PHP Notice: Use of undefined constant Servicegroup - assumed 'Servicegroup' in /usr/local/nagiosxi/html/includes/components/bbmap/bbmap.inc.php on line 194, referer: http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 21 09:04:20.373729 2018] [core:notice] [pid 17907] AH00052: child pid 28459 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374851 2018] [core:notice] [pid 17907] AH00052: child pid 28396 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374915 2018] [core:notice] [pid 17907] AH00052: child pid 28424 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:21.374928 2018] [core:notice] [pid 17907] AH00052: child pid 28425 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:22.376063 2018] [core:notice] [pid 17907] AH00052: child pid 28449 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:22.376125 2018] [core:notice] [pid 17907] AH00052: child pid 28923 exit signal Segmentation fault (11)
[Sun Jan 21 09:04:23.378493 2018] [core:notice] [pid 17907] AH00052: child pid 28974 exit signal Segmentation fault (11)
[Sun Jan 21 08:59:17.335144 2018] [:error] [pid 17914] [client 94.251.35.8:43276] PHP Notice: Array to string conversion in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 469, referer:
http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
[Sun Jan 21 08:59:17.335222 2018] [:error] [pid 17914] [client 94.251.35.8:43276] PHP Notice: Array to string conversion in /usr/local/nagiosxi/html/includes/components/graphexplorer/templates/timeline.inc.php on line 606, referer:
http://nagiosxi.company.com/nagiosxi//includes/page-home-main.php?&=
Everything else seems to be okay. What's going ? Please help.
I have already made
- db restore
- so many times httpd restarts
- full reinstall with db drop (after backup restoring the situation stay the same)
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Mon Jan 22, 2018 3:53 pm
by dwhitfield
IT_dog wrote:
- db restore
- so many times httpd restarts
- full reinstall with db drop (after backup restoring the situation stay the same)
Do you mean db repair when in the restore or are you just saying first you tried it without a drop. Can you attach a screenshot of what you see? Also, please take a look at
https://assets.nagios.com/downloads/nag ... tabase.pdf
Please let us know if you have any trouble with the repair document.
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Mon Jan 22, 2018 11:36 pm
by IT_dog
I have tried both of db repair script and full db reinstall cycle (with deleting mariadb and dropping dbs from disk)
This is my problem:
(
https://imgur.com/a/JCeDs full post)
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Tue Jan 23, 2018 2:58 pm
by dwhitfield
Most of the stuff in the log are notices, which can be ignored. Seeing the error UI, I think the issue may be fixed by changing some PHP settings.
In /etc/php.ini. please change the following to the listed numbers (unless the numbers are higher than these):
max_execution_time = 60
max_input_time = 120
memory_limit = 256M
Also, please add the following line:
max_input_vars = 10000
After making these changes you'll need to restart the httpd service:
If that does not resolve the issue, can you PM me your Profile? You can download it by going to Admin > System Config > System Profile and click the ***Download Profile*** button towards the top. If for whatever reason you *cannot* download the profile, please put the output of View System Info (5.3.4+, Show Profile if older) in the thread (that will at least get us some info). This will give us access to many of the logs we would otherwise ask for individually. If security is a concern, you can unzip the profile take out what you like, and then zip it up again. We may end up needing something you remove, but we can ask for that specifically.
You can also generate a profile manually using the script at /usr/local/nagiosxi/html/includes/components/profile/getprofile.sh
That should generate a profile in /usr/local/nagiosxi/var/components/ which you can get off the server with an application such as FileZilla.
After you PM the profile, please update this thread. Updating this thread is the only way for it to show back up on our dashboard.
If you get an error that PROFILE BUILD FAILED, please see
https://support.nagios.com/kb/article.p ... ategory=44
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Tue Jan 23, 2018 11:09 pm
by IT_dog
Many thanks for the answer.
But while I have tried to find solution ... I lost the problem
Just I don't made any changes and all widgets and dashboards began working as well themselves.
Anyway I've changed php.ini file and I will watch for this strange stuff and keep this topic updated.
Thank you
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Wed Jan 24, 2018 10:39 am
by IT_dog
Well it is stop working again.
That is system info (stripped a little)
Code: Select all
Nagios XI - System Info
System:
Nagios XI Version : 5.4.11
nagiosxi 3.10.0-514.16.1.el7.x86_64 x86_64
CentOS Linux release 7.3.1611 (Core)
Gnome is not installed
Apache Information
PHP Version: 5.4.16
Server Name: nagiosxi.xxx.com
Server Address: x.x.x.x
Server Port: 80
Nagios XI Data
License ends in: NTPRTO
nagios (pid 5715) is running...
NPCD running (pid 17931).
ndo2db (pid 17951) is running...
CPU Load 15: 0.39
Total Hosts: 56
Total Services: 941
Function 'get_base_uri' returns: http://nagiosxi.xxx.com/nagiosxi/
Function 'get_base_url' returns: http://nagiosxi.xxx.com/nagiosxi/
Function 'get_backend_url(internal_call=false)' returns: http://nagiosxi.xxx.com/nagiosxi/includes/components/profile/profile.php
Function 'get_backend_url(internal_call=true)' returns: http://localhost/nagiosxi/backend/
Ping Test localhost
Running:
/bin/ping -c 3 localhost 2>&1
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.031 ms
--- localhost ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.031/0.043/0.059/0.012 ms
Test wget To localhost
WGET From URL: http://localhost/nagiosxi/includes/components/ccm/
Running:
/usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/
--2018-01-24 10:36:34-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://localhost/nagiosxi/login.php?redirect=/nagiosxi/includes/components/ccm/index.php%3f&noauth=1 [following]
--2018-01-24 10:36:34-- http://localhost/nagiosxi/login.php?redirect=/nagiosxi/includes/components/ccm/index.php%3f&noauth=1
Reusing existing connection to [localhost]:80.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: '/usr/local/nagiosxi/tmp/ccm_index.tmp'
0K .......... .......... ..... 1.78M=0.01s
2018-01-24 10:36:34 (1.78 MB/s) - '/usr/local/nagiosxi/tmp/ccm_index.tmp' saved [26247]
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Wed Jan 24, 2018 10:40 am
by IT_dog
I can't send you zip with system dump because it contain a lot of private info.
So any system information can be provided on demand ofc
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Wed Jan 24, 2018 10:55 am
by dwhitfield
Is it possible that you can just delete the two tar.gz files in the profile and then zip up the actual logs? It's going to be very hard to resolve this without getting a complete picture of the server. Do you have any hardening on the system? Can you turn that off for troubleshooting purposes? It's well known that SELinux, modifying sudoers, and changing PAM settings for the system users XI uses causes problems. If we know what was changed we may be able to help you work around it, but ultimately, hardening will be up to you.
UPDATE: profile received and shared with techs
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Wed Jan 24, 2018 1:03 pm
by IT_dog
Sent some stripped info from exported profile.zip
Re: Web UI stop working! ERR_EMPTY_RESPONSE
Posted: Wed Jan 24, 2018 3:40 pm
by dwhitfield
Please run through the following in order and let me know if you run into any issues.
NOTE: You ***must*** use mariadb instead of mysqld in the commands below, ***if*** you have mariadb.
# service nagios stop
# service ndo2db stop
# service mysqld stop
# service crond stop
# service httpd stop
# killall -9 nagios
# killall -9 ndo2db
# rm -f /usr/local/nagios/var/rw/nagios.cmd
# rm -f /usr/local/nagios/var/nagios.lock
# rm -f /usr/local/nagios/var/ndo.sock
# rm -f /usr/local/nagios/var/ndo2db.lock
# rm -f /usr/local/nagiosxi/var/reconfigure_nagios.lock
# for i in `ipcs -q | grep nagios |awk '{print $2}'`; do ipcrm -q $i; done
# service mysqld start
# service ndo2db start
# service nagios start
# service httpd start
# service crond start
You definitely have kernel queue issues, so you'll need to increase the settings *LIKE* those in
https://support.nagios.com/kb/article.php?id=139
I want to emphasize that the numbers in that document are conservative. If the numbers in the document do not resolve the issue, you may need to increase them more.