Reports Loading Server

[jbennett]

I've been running into issues when trying to use the reporting feature of Nagios XI (1.3). We are looking to generate montly reports for various host groups, but I'm concerned about the server not being able to process these reports. Just to process a report for a host group with XX hosts for the previous month, it takes over 10 minutes (many times, Chrome and IE will tell me that the page has timed out), with process information as follows:

Sitting at the Nagios home page:

Code: Select all

top - 20:56:12 up 3 min,  1 user,  load average: 9.46, 4.75, 1.86
Tasks: 284 total,   1 running, 283 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.6%us,  1.3%sy,  0.0%ni, 32.6%id, 65.3%wa,  0.0%hi,  0.2%si,  0.0%st
Mem:  15463692k total,  1161008k used, 14302684k free,    66596k buffers
Swap:   262136k total,        0k used,   262136k free,   283960k cached

Accessing the reports tab:

Code: Select all

top - 20:58:40 up 6 min,  1 user,  load average: 10.39, 6.79, 3.04
Tasks: 276 total,   2 running, 274 sleeping,   0 stopped,   0 zombie
Cpu(s): 16.3%us,  2.1%sy,  0.0%ni, 61.4%id, 19.7%wa,  0.0%hi,  0.4%si,  0.0%st
Mem:  15463692k total,  1300388k used, 14163304k free,    76900k buffers
Swap:   262136k total,        0k used,   262136k free,   332828k cached

Running a report on a host group for the previous month:

Code: Select all

top - 21:02:39 up 10 min,  1 user,  load average: 13.80, 9.66, 5.06
Tasks: 248 total,   2 running, 245 sleeping,   0 stopped,   1 zombie
Cpu(s): 32.0%us,  9.2%sy,  0.0%ni, 43.9%id, 12.0%wa,  0.1%hi,  2.8%si,  0.0%st
Mem:  15463692k total,  1297216k used, 14166476k free,    82180k buffers
Swap:   262136k total,        0k used,   262136k free,   409704k cached

The report i'm asking Nagios to run is for 58 hosts.

All told, my nagios monitoring performance is as follows currently:

Code: Select all

 Monitoring Performance
Service Check Execution Time:	0.01 / 10.20 / 0.667 sec
Service Check Latency:	0.00 / 260.52 / 12.881 sec
Host Check Execution Time:	0.00 / 10.13 / 0.248 sec
Host Check Latency:	0.00 / 55.87 / 10.923 sec
# Active Host / Service Checks:	3214 / 4311
# Passive Host / Service Checks:	0 / 0

Is this to be expected on such an instance of Nagios?

[jbennett]

I have started to notice another issue. When I try and access the reporting section, I do not have the option to drill down to specific hosts, host groups, or services. The fields are there, but there is nothing being populated in those drop down boxes. When I click the drop down, I only get the following as options:

Host:
Hostgroup:
Servicegroup:

[mguthrie]

The first thing I would check on this is database corruption. If you're using a VM be sure to avoid hard power-offs, mysql doesn't react well to them.

http://assets.nagios.com/downloads/nagi ... tabase.pdf

[jbennett]

I have run the database repair. Still getting long waits to just access the reports page and once I'm there, I do not get anything additional in the drop down.

The secondary issue of items not being listed in the drop downs, didn't seem to happen prior to my activating the 60-day enterprise trial.

[scottwilkerson]

What I noticed from looking at your first post you have a ton of I/O wait time

Code: Select all

65.3%wa

This could be related to a failing disk, but also could be tightly tied to the database. Can you run the following commands and report the output

Code: Select all

mysql -u root -p'nagiosxi' -e "SHOW PROCESSLIST;"|wc -l
cat /etc/my.cnf |grep max_connections

Then I would also run the following to make sure everything is running properly

Code: Select all

service nagios stop
service ndo2db stop
killall -9 nagios
killall -9 ndo2db
service ndo2db start
service nagios start
service npcd restart

Finally, how many CPU cores does this machine have

[jbennett]

Code: Select all

[root@nagiosxivm ~]# cat /etc/my.cnf |grep max_connections
max_connections=300

[slansing]

Have you ran the other commands yet? If so, what was the result after checking again?

[jbennett]

Ahh, I guess I didn't realize I needed to check it again. It is the same when running immediately after starting/stopping all of the services you listed.

You mentioned long IO times. I'm reading up on utilizing a RAM disk, would this help? It is possible that the previous admin has already done this, but I'm not sure how to know for sure.

I don't seem to have a /usr/local/nagios/var/status.dat file, only the objects.cache (3.9MB) file suggested in this documentation.

This is on a VM w/ 4 - 2 core processors (8 total processors - max supported) and 16GB of memory.

[slansing]

Yes integrating a Ram Disk as well as offloading the MySQL database to another location would greatly help with disk IO, in addition to this integrating Mod Gearman will take a huge chunk as well. Though with your system specs I don't think this is the issue we are seeing.

Offloading the MySQL database: http://assets.nagios.com/downloads/nagi ... Server.pdf

Mod Gearman: http://assets.nagios.com/downloads/nagi ... ios_XI.pdf