WEB UI becomes unresponsive after config apply

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
Locked
gdolidze
Posts: 154
Joined: Tue Apr 07, 2015 10:07 am

WEB UI becomes unresponsive after config apply

Post by gdolidze »

We updated our 64bit vm from 5.2.* to 5.4.1 and the web interface started lagging became really slow, sometimes its even unresponsive.
We did mv /usr/local/nagiosxi/html/includes/components/nagiosim /tmp/ and that fixed the issue.
Now we notice that when we edit a service and do a apply configuration, the WEB UI becomes unresponsive for around 5 mins then goes back to normal.
Before we did the upgrade it took a couple of seconds.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: WEB UI becomes unresponsive after config apply

Post by mcapra »

Can you PM me a system profile for us to look at?

Also, try to run an Apply Configuration while running a tail on the following file (share its output):

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Also, the contents of the following files after an apply config might be helpful:

Code: Select all

/var/log/httpd/access_log
/var/log/httpd/error_log
/var/log/httpd/ssl_access_log
/var/log/httpd/ssl_error_log
EDIT: profile received and placed in shared space for techs
Former Nagios employee
https://www.mcapra.com/
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: WEB UI becomes unresponsive after config apply

Post by mcapra »

Got the profile, though it seems to be a bit dated (messages from Jan 30). Can I still have you do the following:
mcapra wrote: Also, try to run an Apply Configuration while running a tail on the following file (share its output):

Code: Select all

tail -f /usr/local/nagiosxi/var/cmdsubsys.log
Former Nagios employee
https://www.mcapra.com/
Top
gdolidze
Posts: 154
Joined: Tue Apr 07, 2015 10:07 am

Re: WEB UI becomes unresponsive after config apply

Post by gdolidze »

the cmdsubsys file is just writing this over and over again

PROCESSED 1 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........................................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
.
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........................................................
PROCESSED 0 COMMANDS
tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated
...........................................................tail: /usr/local/nagiosxi/var/cmdsubsys.log: file truncated




The SSL_access file is empty and the access file is too big so here is the last 2 pages.

10.1.3.145 - - [31/Jan/2017:14:27:27 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getcommandstatus&opts=5023&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 121 "http://10.1.1.9/nagiosxi/includes/compo ... config.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_host_status_summary_html%22%2C%22args%22%3A%7B%7D%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 1864 "http://10.1.1.9/nagiosxi//includes/page ... splash.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_component_states_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 4983 "http://10.1.1.9/nagiosxi/admin/main.php?&=" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=keepalive&opts=6&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 - "http://10.1.1.9/nagiosxi/includes/compo ... -index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_admin_tasks_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 901 "http://10.1.1.9/nagiosxi/admin/main.php?&=" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 660 "http://10.1.1.9/nagiosxi/includes/compo ... -index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:27 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 855 "http://10.1.1.9/nagiosxi/includes/compo ... -index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 855 "http://10.1.1.9/nagiosxi/includes/compo ... -index.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_pagetop_alert_content_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 660 "http://10.1.1.9/nagiosxi/admin/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_tray_alert_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 855 "http://10.1.1.9/nagiosxi/admin/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:42 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getcommandstatus&opts=5023&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 121 "http://10.1.1.9/nagiosxi/includes/compo ... config.php" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:58 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_component_states_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 4984 "http://10.1.1.9/nagiosxi/admin/main.php?&=" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
10.1.3.145 - - [31/Jan/2017:14:27:58 -0500] "GET /nagiosxi/ajaxhelper.php?cmd=getxicoreajax&opts=%7B%22func%22%3A%22get_component_states_html%22%2C%22args%22%3A%22%22%7D&nsp=b7caabeeeacf3cd3bfad5a0ed30910c3f703b91abcafac6fd23077aedca2d597 HTTP/1.1" 200 4984 "http://10.1.1.9/nagiosxi/admin/main.php?&=" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
You do not have the required permissions to view the files attached to this post.
Top
rkennedy
Posts: 6579
Joined: Mon Oct 05, 2015 11:45 am

Re: WEB UI becomes unresponsive after config apply

Post by rkennedy »

Does the Apply Configuration finish successfully, or does it simply loop? The cmdsubsys doesn't seem to have a log of anything happening which is concerning.

I did see this in your error_log -

Code: Select all

[Tue Jan 31 14:27:11 2017] [error] [client 10.1.3.145] connect() timed out!, referer: http://10.1.1.9/nagiosxi/
[Tue Jan 31 14:27:11 2017] [error] [client 10.1.3.145] connect() timed out!, referer: http://10.1.1.9/nagiosxi/
[Tue Jan 31 14:27:12 2017] [error] [client 10.1.3.145] connect() timed out!, referer: http://10.1.1.9/nagiosxi/includes/components/nagioscorecfg/applyconfig.php
[Tue Jan 31 14:27:27 2017] [error] [client 10.1.3.145] connect() timed out!, referer: http://10.1.1.9/nagiosxi/includes/components/ccm/xi-index.php
Former Nagios Employee
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: WEB UI becomes unresponsive after config apply

Post by mcapra »

It looks like the apply configuration step is never submit to the command subsystem. Either that or the command being run is timing out or stuck in a loop.

Can you share the output of the following commands executed from the CLI of your Nagios XI machine:

Code: Select all

/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
cd /tmp && /usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/ && ls -al
Former Nagios employee
https://www.mcapra.com/
Top
gdolidze
Posts: 154
Joined: Tue Apr 07, 2015 10:07 am

Re: WEB UI becomes unresponsive after config apply

Post by gdolidze »

[root@nagios tmp]# cd /tmp && /usr/bin/wget http://localhost/nagiosxi/includes/components/ccm/ && ls -al
--2017-02-03 14:08:14-- http://localhost/nagiosxi/includes/components/ccm/
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://localhost/nagiosxi/login.php?red ... f&noauth=1 [following]
--2017-02-03 14:08:14-- http://localhost/nagiosxi/login.php?red ... f&noauth=1
Connecting to localhost|::1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “index.html.2”

[ <=> ] 26,045 --.-K/s in 0.01s

2017-02-03 14:08:14 (1.72 MB/s) - “index.html.2” saved [26045]


-rw------- 1 nagios nagios 0 Nov 13 03:25 ZZZHOT
You do not have the required permissions to view the files attached to this post.
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: WEB UI becomes unresponsive after config apply

Post by mcapra »

Can you run the following commands and share their outputs:

Code: Select all

chage -l nagios
chage -l apache
cat /etc/passwd | grep apache
cat /etc/passwd | grep nagios

su nagios
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
exit

ls -al /usr/local/nagiosxi/scripts/
getenforce
Former Nagios employee
https://www.mcapra.com/
Top
gdolidze
Posts: 154
Joined: Tue Apr 07, 2015 10:07 am

Re: WEB UI becomes unresponsive after config apply

Post by gdolidze »

Code: Select all

[root@nagios ~]# chage -l nagios
Last password change                                    : Feb 24, 2015
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : 0
Maximum number of days between password change          : 99999
Number of days of warning before password expires       : 7


[root@nagios ~]# chage -l apache
Last password change                                    : Feb 24, 2015
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change           : -1
Maximum number of days between password change          : -1
Number of days of warning before password expires           : -1


apache:x:48:48:Apache:/var/www:/sbin/nologin
nagios:x:500:100::/home/nagios:/bin/bash

for su nagios
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
this is repeated

Code: Select all

PHP Warning:  fgets() expects parameter 1 to be resource, boolean given in /usr/local/nagiosxi/scripts/nagiosql_login.php on line 51
PHP Warning:  feof() expects parameter 1 to be resource, boolean given in /usr/local/nagiosxi/scripts/nagiosql_login.php on line 49
PHP Warning:  fgets() expects parameter 1 to be resource, boolean given in /usr/local/nagiosxi/scripts/nagiosql_login.php on line 51
PHP Warning:  feof() expects parameter 1 to be resource, boolean given in /usr/local/nagiosxi/scripts/nagiosql_login.php on line 49


drwxr-xr-x  2 nagios nagios   4096 Feb  4 01:00 .
drwxr-xr-x 10 nagios nagios   4096 Feb 24  2015 ..
-rwxr-x---  1 nagios nagios   5673 Jan 30 09:38 backup_xi.sh
-rwxr-x---  1 nagios nagios   1664 Jan 30 09:38 change_timezone.sh
-rw-r-----  1 nagios nagios   3030 Jan 30 09:38 contact_notification_handler.php
-rwxr-x---  1 nagios nagios    660 Jan 30 09:38 export_nagiosql.sh
-rw-r--r--  1 nagios nagios     58 Jan 30 09:38 .gitignore
-rwxr-x---  1 nagios nagios    261 Jan 30 09:38 handle_nagioscore_event.php
-rw-r-----  1 nagios nagios   3318 Jan 30 09:38 handle_nagioscore.inc.php
-rwxr-x---  1 nagios nagios    262 Jan 30 09:38 handle_nagioscore_notification.php
-rwxr-x---  1 nagios nagios    425 Jan 30 09:38 import_nagiosql.sh
-rwxr-x---  1 nagios nagios    908 Jan 30 09:38 import_xiconfig.php
-rwxr-x---  1 nagios nagios     93 Jan 30 09:38 kill_rrdtool.sh
-rwxr-x---  1 nagios nagios   2303 Jan 30 09:38 manage_services.sh
-rwxrwxr-x  1 nagios nagios    161 Jan 31 13:58 nagiosql.cookies
-rwxrwxr-x  1 nagios nagios 283701 Jan 30 09:38 nagiosql_defaults.sql
-rwxrwxr-x  1 nagios nagios    895 Jan 30 09:38 nagiosql_delete_contact.php
-rwxrwxr-x  1 nagios nagios  39878 Aug 26 15:57 nagiosql.delete.host
-rwxrwxr-x  1 nagios nagios   1937 Jan 30 09:38 nagiosql_delete_host.php
-rwxrwxr-x  1 nagios nagios    497 Jan 30 09:38 nagiosql_delete_object.sh
-rwxrwxr-x  1 nagios nagios  47676 Nov  4 15:58 nagiosql.delete.service
-rwxrwxr-x  1 nagios nagios   2766 Jan 30 09:38 nagiosql_delete_service.php
-rwxrwxr-x  1 nagios nagios    896 Jan 30 09:38 nagiosql_delete_timeperiod.php
-rwxrwxr-x  1 nagios nagios   1272 Jan 30 09:38 nagiosql_exportall.php
-rwxrwxr-x  1 nagios nagios  21683 Jan 31 13:58 nagiosql.export.monitoring
-rwxrwxr-x  1 nagios nagios   1296 Jan 30 09:38 nagiosql_importall.php
-rwxrwxr-x  1 nagios nagios  38407 Jan 31 11:27 nagiosql.import.monitoring
-rwxrwxr-x  1 nagios nagios  37626 Jan 31 13:58 nagiosql.login
-rwxrwxr-x  1 nagios nagios   1681 Jan 30 09:38 nagiosql_login.php
-rwxrwxr-x  1 nagios nagios   2247 Jan 30 09:38 nagiosql_snapshot.sh
-rwxrwxr-x  1 nagios nagios    200 Jan 30 09:38 nagiosql_trim_backups.sh
-rw-r-----  1 nagios nagios    366 Jan 30 09:38 nagiosxi_dbtype.php
-rwxr-x---  1 nagios nagios    408 Jan 30 09:38 nom_create_nagioscore_checkpoint_cond.sh
-rwxr-x---  1 nagios nagios    961 Jan 30 09:38 nom_create_nagioscore_checkpoint.sh
-rwxr-x---  1 nagios nagios    774 Jan 30 09:38 nom_create_nagioscore_errorpoint.sh
-rwxr-x---  1 nagios nagios    760 Jan 30 09:38 nom_restore_nagioscore_checkpoint.sh
-rwxr-x---  1 nagios nagios    802 Jan 30 09:38 nom_restore_nagioscore_checkpoint_specific.sh
-rwxr-x---  1 nagios nagios   2893 Jan 30 09:38 nom_trim_nagioscore_checkpoints.sh
-rwxr-x---  1 nagios nagios   4621 Jan 30 09:38 parse_core_eventlog.php
-rwxr-x---  1 nagios nagios   6019 Jan 30 09:38 patch_ndoutils.php
-rwxr-x---  1 nagios nagios    588 Jan 30 09:38 reconfigure_nagios.sh
-rwxr-x---  1 nagios nagios   1378 Jan 30 09:38 repair_databases.sh
-rwxr-x---  1 nagios nagios   3315 Jan 30 09:38 repairmysql.sh
-rwxr-x---  1 nagios nagios   6798 Apr 29  2015 reset_config_perms
-rw-r-----  1 nagios nagios    280 Apr 29  2015 reset_config_perms.c
-rwsr-xr-x  1 root   nagios   1010 Jan 30 09:38 reset_config_perms.sh
-rwxr-x---  1 nagios nagios   1538 Jan 30 09:38 reset_nagiosadmin_password.php
-rwxr-x---  1 nagios nagios   1591 Jan 30 09:38 restart_nagios_with_export.sh
-rwxr-x---  1 nagios nagios   3460 Jan 30 09:38 restore_defaults.sh
-rwxr-x---  1 nagios nagios   9031 Jan 30 09:38 restore_xi.sh
-rwxr-x---  1 nagios nagios   1847 Jan 30 09:38 send_to_auditlog.php
-rw-r-----  1 nagios nagios   2702 Jan 30 09:38 uninstall_xi.sh
-rwxr-x---  1 nagios nagios   1207 Jan 30 09:38 unlock_user_account.php
-rwxr-x---  1 nagios nagios   2681 Jan 30 09:38 upgrade_to_latest.sh

[root@nagios ~]# getenforce
Disabled
Last edited by tmcdonald on Tue Feb 07, 2017 3:14 pm, edited 1 time in total.
Reason: Please use [code][/code] tags around long output
Top
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: WEB UI becomes unresponsive after config apply

Post by mcapra »

That nagiosql.login file looks pretty old. There might be a lock on it that's preventing it from being written. Can you run the following:

Code: Select all

rm /usr/local/nagiosxi/scripts/nagiosql.login
Then re-run the following commands:

Code: Select all

su nagios
cd /usr/local/nagiosxi/scripts/
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
exit

ls -al /usr/local/nagiosxi/scripts/
If that produces the same errors, please edit the /usr/local/nagiosxi/scripts/nagiosql_login.php file around line 46 from this:

Code: Select all

//login verification for nagiosql
$f = @fopen('nagiosql.login','r');
$check = false;
To this:

Code: Select all

//login verification for nagiosql
$f = @fopen('nagiosql.login','r');
error_get_last();
var_dump($f);
$check = false;
Then re-run the commands and share their output:

Code: Select all

su nagios
cd /usr/local/nagiosxi/scripts/
/usr/local/nagiosxi/scripts/reconfigure_nagios.sh
exit

ls -al /usr/local/nagiosxi/scripts/
Former Nagios employee
https://www.mcapra.com/
Top
Locked

Return to “Nagios XI”