Host and service detail pages taking >2 mins to load

This support forum board is for support questions relating to Nagios XI, our flagship commercial network monitoring solution.
matt_ps
Posts: 11
Joined: Mon Oct 09, 2017 11:50 am

Host and service detail pages taking >2 mins to load

Post by matt_ps »

The basics: the instance is hosted in AWS, used for externally monitoring our infrastructure. CentOS 7 x64 VM with Nagios XI installed. Using SSL to view the site.

The issue: when drilling down into a service or host, and getting to the part where it allows you to view the status details, force an immediate recheck, acknowledge an alarm, etc., the page loads take 1-2 minutes. This started a couple weeks ago. I attempted to log onto the web portal to configure a new monitor and it gave me an alert saying to run /usr/local/nagiosxi/scripts/repair_databases.sh. I received the following output:

===============
REPAIR COMPLETE
===============

=======================
nagios database repair succeeded
nagiosql database repair succeeded
nagiosxi database repair succeeded



After running that script and was able to get in to the interface, but then I noticed that the monitors were having this issue of incredibly slow load times.

I came across this thread and began running through the things suggested (https://support.nagios.com/forum/viewto ... 16&t=39176). I set the internal program URL to its inside IP on the AWS instance, and left the external program URL with the host name that I'm accessing the instance with remotely. That had no change.

I then checked /var/log/mariadb.log and found the following:

Code: Select all

171031 08:39:24 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
171031  8:39:31 [Note] /usr/libexec/mysqld (mysqld 5.5.56-MariaDB) starting as process 2147 ...
171031  8:39:33 InnoDB: The InnoDB memory heap is disabled
171031  8:39:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
171031  8:39:33 InnoDB: Compressed tables use zlib 1.2.7
171031  8:39:33 InnoDB: Using Linux native AIO
171031  8:39:34 InnoDB: Initializing buffer pool, size = 128.0M
171031  8:39:34 InnoDB: Completed initialization of buffer pool
171031  8:39:34 InnoDB: highest supported file format is Barracuda.
171031  8:39:37  InnoDB: Waiting for the background threads to start
171031  8:39:38 Percona XtraDB (http://www.percona.com) 5.5.52-MariaDB-38.3 started; log sequence number 1633177
171031  8:39:38 [Note] Plugin 'FEEDBACK' is disabled.
171031  8:39:39 [Note] Server socket created on IP: '0.0.0.0'.
171031  8:39:44 [Note] Event Scheduler: Loaded 0 events
171031  8:39:44 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.56-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
171031  8:39:49 [ERROR] mysqld: Table './nagios/nagios_conninfo' is marked as crashed and should be repaired
171031  8:39:49 [Warning] Checking table:   './nagios/nagios_conninfo'
171031  8:39:51 [ERROR] mysqld: Table './nagios/nagios_programstatus' is marked as crashed and should be repaired
171031  8:39:51 [Warning] Checking table:   './nagios/nagios_programstatus'
171031  8:39:51 [ERROR] mysqld: Table './nagios/nagios_hoststatus' is marked as crashed and should be repaired
171031  8:39:51 [Warning] Checking table:   './nagios/nagios_hoststatus'
171031  8:39:51 [ERROR] mysqld: Table './nagios/nagios_servicestatus' is marked as crashed and should be repaired
171031  8:39:51 [Warning] Checking table:   './nagios/nagios_servicestatus'
171031  8:39:51 [ERROR] mysqld: Table './nagios/nagios_logentries' is marked as crashed and should be repaired
171031  8:39:51 [Warning] Checking table:   './nagios/nagios_logentries'
171031  8:40:01 [ERROR] mysqld: Table './nagios/nagios_systemcommands' is marked as crashed and should be repaired
171031  8:40:01 [Warning] Checking table:   './nagios/nagios_systemcommands'
171031  8:40:01 [ERROR] mysqld: Table './nagios/nagios_eventhandlers' is marked as crashed and should be repaired
171031  8:40:01 [Warning] Checking table:   './nagios/nagios_eventhandlers'
171031  8:40:01 [ERROR] mysqld: Table './nagios/nagios_customvariablestatus' is marked as crashed and should be repaired
171031  8:40:01 [Warning] Checking table:   './nagios/nagios_customvariablestatus'
171031  8:40:09 [ERROR] mysqld: Table './nagiosxi/xi_sysstat' is marked as crashed and should be repaired
171031  8:40:09 [ERROR] mysqld: Table './nagiosxi/xi_meta' is marked as crashed and should be repaired
171031  8:40:09 [Warning] Checking table:   './nagiosxi/xi_sysstat'
171031  8:40:09 [ERROR] mysqld: Table './nagiosxi/xi_eventqueue' is marked as crashed and should be repaired
171031  8:40:09 [Warning] Checking table:   './nagiosxi/xi_meta'
171031  8:40:10 [Warning] Checking table:   './nagiosxi/xi_eventqueue'
171031  8:40:12 [ERROR] mysqld: Table './nagios/nagios_notifications' is marked as crashed and should be repaired
171031  8:40:12 [Warning] Checking table:   './nagios/nagios_notifications'
171031  8:40:12 [ERROR] mysqld: Table './nagios/nagios_contactnotifications' is marked as crashed and should be repaired
171031  8:40:12 [Warning] Checking table:   './nagios/nagios_contactnotifications'
171031  8:40:12 [ERROR] mysqld: Table './nagios/nagios_contactnotificationmethods' is marked as crashed and should be repaired
171031  8:40:12 [Warning] Checking table:   './nagios/nagios_contactnotificationmethods'
171031  8:40:12 [ERROR] mysqld: Table './nagios/nagios_statehistory' is marked as crashed and should be repaired
171031  8:40:12 [Warning] Checking table:   './nagios/nagios_statehistory'
171031  8:40:13 [ERROR] mysqld: Table './nagiosxi/xi_events' is marked as crashed and should be repaired
171031  8:40:13 [Warning] Checking table:   './nagiosxi/xi_events'
After seeing this, I ran the repair_databases.sh again, and now am no longer seeing these at the end of the log, but the page loads are still happening at the same >2 minutes per page load.
Last edited by Anonymous on Wed Nov 08, 2017 1:03 pm, edited 2 times in total.
Reason: Please use [code][/code] tags around long output
kyang

Re: Host and service detail pages taking >2 mins to load

Post by kyang »

If the host and service detail pages are taking a long time to load. Can you post your apache error_logs

Code: Select all

tail -f /var/log/httpd/error_log
matt_ps
Posts: 11
Joined: Mon Oct 09, 2017 11:50 am

Re: Host and service detail pages taking >2 mins to load

Post by matt_ps »

Code: Select all

[Sun Nov 05 03:07:03.235265 2017] [auth_digest:notice] [pid 745] AH01757: generating secret for digest authentication ...
[Sun Nov 05 03:07:03.242508 2017] [lbmethod_heartbeat:notice] [pid 745] AH02282: No slotmem from mod_heartmonitor
[Sun Nov 05 03:07:03.278638 2017] [mpm_prefork:notice] [pid 745] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Sun Nov 05 03:07:03.278655 2017] [core:notice] [pid 745] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 06 08:08:16.864024 2017] [mpm_prefork:notice] [pid 745] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Nov 06 08:09:33.254829 2017] [suexec:notice] [pid 6567] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 06 08:09:33.281803 2017] [auth_digest:notice] [pid 6567] AH01757: generating secret for digest authentication ...
[Mon Nov 06 08:09:33.282364 2017] [lbmethod_heartbeat:notice] [pid 6567] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 06 08:09:33.322055 2017] [mpm_prefork:notice] [pid 6567] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon Nov 06 08:09:33.322083 2017] [core:notice] [pid 6567] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 06 09:16:06.978430 2017] [mpm_prefork:notice] [pid 6567] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Nov 06 09:16:09.067302 2017] [suexec:notice] [pid 30235] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 06 09:16:09.093321 2017] [auth_digest:notice] [pid 30235] AH01757: generating secret for digest authentication ...
[Mon Nov 06 09:16:09.093879 2017] [lbmethod_heartbeat:notice] [pid 30235] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 06 09:16:09.133592 2017] [mpm_prefork:notice] [pid 30235] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon Nov 06 09:16:09.133627 2017] [core:notice] [pid 30235] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 06 09:21:35.274112 2017] [mpm_prefork:notice] [pid 30235] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Nov 06 09:21:58.382406 2017] [suexec:notice] [pid 32415] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 06 09:21:58.407439 2017] [auth_digest:notice] [pid 32415] AH01757: generating secret for digest authentication ...
[Mon Nov 06 09:21:58.407995 2017] [lbmethod_heartbeat:notice] [pid 32415] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 06 09:21:58.447844 2017] [mpm_prefork:notice] [pid 32415] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon Nov 06 09:21:58.447874 2017] [core:notice] [pid 32415] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Mon Nov 06 09:22:16.746248 2017] [mpm_prefork:notice] [pid 32415] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Nov 06 09:22:22.806076 2017] [suexec:notice] [pid 564] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Mon Nov 06 09:22:22.831126 2017] [auth_digest:notice] [pid 564] AH01757: generating secret for digest authentication ...
[Mon Nov 06 09:22:22.831714 2017] [lbmethod_heartbeat:notice] [pid 564] AH02282: No slotmem from mod_heartmonitor
[Mon Nov 06 09:22:22.870581 2017] [mpm_prefork:notice] [pid 564] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 configured -- resuming normal operations
[Mon Nov 06 09:22:22.870608 2017] [core:notice] [pid 564] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

This doesn't seem to show my page loads that were around 15:00 local server time... Not sure if that is significant or not.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Host and service detail pages taking >2 mins to load

Post by npolovenko »

@matt_ps, Not too much information from that log. Could you perhaps upload your whole Nagios XI System Profile so I can review it?
To send us your system profile. Login to the Nagios XI GUI using a web browser.
Click the "Admin" > "System Profile" Menu
Click the "Download Profile" button
Save the profile.zip file and upload it here or send it in a personal message. If the file is too big for pm you could upload it to the cloud storage and send me the link.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
matt_ps
Posts: 11
Joined: Mon Oct 09, 2017 11:50 am

Re: Host and service detail pages taking >2 mins to load

Post by matt_ps »

I've sent this via PM.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Host and service detail pages taking >2 mins to load

Post by npolovenko »

@matt_ps, Let's try increasing the npcd resource load threshold on your XI server:
Open the npcd config file:
/usr/local/nagios/etc/pnp/npcd.cfg
And change this line:
load_threshold = 10.0
to this:
load_threshold = 20.0
Save and exit

When that is done please restart npcd daemon:

Code: Select all

service npcd restart
and please restart Nagios and httpd as well:

Code: Select all

service nagios restart
service httpd restart
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
matt_ps
Posts: 11
Joined: Mon Oct 09, 2017 11:50 am

Re: Host and service detail pages taking >2 mins to load

Post by matt_ps »

I have performed those steps and there was no change in the page load time. Chrome debugger shows that it took 2.1 minutes to load.

EDIT: I'm noticing that the screenshot I took yesterday, and the screenshot I took today of Chrome debugger both are 2.1 minutes. I'm wondering if that's a clue. I've attached the two screenshots to this post.

EDIT 2: I've ran more tests on more services and hosts, and they each load in 2.1 minutes...
You do not have the required permissions to view the files attached to this post.
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Host and service detail pages taking >2 mins to load

Post by npolovenko »

@matt_ps, On the XI server, in the hosts file please add the ip address and the hostname of the computer that you're trying to access Nagios XI from.

Code: Select all

/etc/hosts
In a following format:
ip HostName

Also in XI web interface please go to Admin, and then System Settings in the left column. Fill out the External URL field.

After that please restart Nagios:

Code: Select all

service nagios restart
service httpd restart
Let us know if that fixed the problem.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
matt_ps
Posts: 11
Joined: Mon Oct 09, 2017 11:50 am

Re: Host and service detail pages taking >2 mins to load

Post by matt_ps »

Well that host file entry seemed to fix the issue. Any idea why setting public DNS on the External URL causes the application to choke? Shouldn't it only try and access itself using the internal URL anyways?
npolovenko
Support Tech
Posts: 3457
Joined: Mon May 15, 2017 5:00 pm

Re: Host and service detail pages taking >2 mins to load

Post by npolovenko »

@matt_ps, I'm glad that worked for you.
Any idea why setting public DNS on the External URL causes the application to choke? Shouldn't it only try and access itself using the internal URL anyways?
Not if you access it externally. The system needs to know where the request came from and send it to that URL. My guess that this has to do with DNS lookup. When Nagios doesn't know the actual client's IP the return request somehow gets lost. Normally DNS lookup would take place but it didn't for some reason. Hope that made it clearer to you. May i close this thread as resolved?
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked