NagiosXI performance issue

[anoop]

Hi Team,

Actually, we are using 32GB of RAM for Nagios XI Server and presently utilizing 4GB approx for 1800 hosts and 9800 service checks.. Do we need to assign more space for RAM Disk or is there any option to increase RAM Disk Space later...?

Moreover, Can we use "ext4" FS instead of "tmpfs" which is provided in the document..???

Suggest us on above two points...

Thanks in advance

[abrist]

I would almost guarantee that your issues are due to i/o wait.

Do we need to assign more space for RAM Disk or is there any option to increase RAM Disk Space later...?

Make sure your ramdisk is large enough for the rate of checks in your install. Most of our client run their ramdisks from 50mb to 250mb.

Moreover, Can we use "ext4" FS instead of "tmpfs" which is provided in the document..???

tmpfs is much faster than ext4 on disk. This will probably make your i/o wait issue worse.

What type of disk array is this using? If it is a SAN, can you get some statistics on the latency and write speeds?

[anoop]

Hi Team,

Our current NagiosXI server Storage details

Storage Model : Dell
Storage Array Type: SAS
Storage provided: 200GB
IOPS Provided: Assigned unlimited option and value is 1000 IOPS

Please verify the below attachments for the statistics provided for both NagiosXI and Offloaded DB server.


NagiosXi and offloaded server configured in VM environment

[sreinhardt]

A would agree with abrist. It is a little hard to dicern what your docs with tables are showing as the usage isn't labeled. However there are fairly continually high wait times, and spikes in speed on the graphs, which generally is not the case in a nagios environment. Unless checks are falling behind due to load or latency it is usually a fairly stable ups and downs. With that said, are both the nagios server and the mysql server housed on the same hardware, just separated by vm? As abrist mentioned before, are you using a san or any other remote storage, especially since you have a choice of IOPS?

[anoop]

HI Team,

We are using SAN Storage with SAS drives and provided with same hardware for Nagios XI Server and offloaded DB Server.

Last night, we configured RAM Disk and left it for observation, but as per the statistics, we are still facing IOwait issue and didn't change much compared to previous. From SAR Report, the average IOWait is shwoing as 10% ancd sometimes going upto 35%

Sometimes, in "top" command, we are getting some services into "D" State . Please find the attachment for your reference.

If i "Apply Configuration", it is taking more time like hours & hours..

Please suggest us with better solution.

[anoop]

Hi team,

Is there anything we need to change from the performance settings tab in Nagios Xi server.

Please check the below attachments where we are using default options and let us know if any changes required for better performance..

And checked with OS team and also with storage team and we got output like iowait is showing 10% and it is in normal state.. using SAN Storage with SAS Drives..

Please let us know if we need to perform any performance tuning..

Thanks in advance..

[abrist]

Did you ever get latency times for your san? The problem is i/o wait and that is a hardware specific, environment related issue. 10% to 35% is way too high. At a minimum of 8000 files, your disk write/read latency could cause the apply to take minutes. As you claim it is taking hours, there could be other issues as well. Have you increased the limits in php.ini at all?

[anoop]

HI team,

Please find the below attachment of my php.ini file for your reference..

And at the time it is taking hours to scan a device or appply configuration, im just rebooting my server and later it is working fine within 10 minutes...

[sreinhardt]

Did you ever get latency times for your san? The problem is i/o wait and that is a hardware specific, environment related issue. 10% to 35% is way too high. At a minimum of 8000 files, your disk write/read latency could cause the apply to take minutes.

As for your claim that it is taking hours, this is not possible with your current settings. The max execution time on php.ini, which you just showed, is set to 1200 seconds. This turns out to be 20 minutes, which is admittedly a long time. In addition to apply config taking a long time, how is doing the steps individually from the write config tool in CCM? Is it possible that you have circular host or service groups\templates?

[anoop]

Hi Team,

Please check and verify attached graphs for cpu, memory and disk usage of NagiOS XI server. Graphs generated from VMWare infrastructure team, where this NagiOS XI Server deployed.

Please let us know what steps we need to be taken for resolving this issue.