I/O wait issue

[snapon_admin]

Noticing a high percentage of I/O wait on our NNA server and wondering what options I have to reduce it. Our server has ~100-120 sources, 8 CPU cores (pretty sure it's 2 quad cores but it might be 4 dual cores), 8GB RAM, using the vSphere OVF template. This is what I'm looking at:

IO wait NNA.png

What do?

[scottwilkerson]

Does this machine share drives with any other machines (e.g. SAN or VM's)?

Does the server have spinning drives or SSD's?

[snapon_admin]

Ahh, good thought. I'll have to ask our VM admins as they own it, but I'd be willing to bet it's sharing drive space with other VMs and that it's likely not running on SSDs. I will check and get back to you. Not sure why I didn't think of the most obvious thing first....thanks!

[cmerchant]

Let us know what you find out. Thanks.

[snapon_admin]

Looks like, as I suspected, our VMs are all on shared SAN storage. I don't see it being possible for that to be changed, so is there anything I can do to bring the I/O wait down at all?

[sreinhardt]

Are you able to correlate those spikes to anything in particular? Maybe when reports are being run, heavy user usage, maybe spikes in network traffic being pumped to NNA? The seem to be for relatively short (5-15 minute) windows and only when they overlap is it really hard on your system for longer periods of time.

[snapon_admin]

Well I disabled the Network analyzer tab in XI, the one that shows up when you click on a host or service, and ever since the I/O spikes have almost completely disappeared. I'm assuming people were click on it, possibly not knowing what it even did, and since that tab runs a query against ALL sources on the NNA server it was causing the issue. Ever since I disabled the tab, there has still been a load spike at almost the same time both days. At around 8-8:30 am yesterday and the day before (after I disabled the tab) the I/O wait shoots up to about 30ish for an hour. After that all is well.

[sreinhardt]

That makes a lot of sense as to why the increased load. Seems like around the time people start coming into the office it spikes some, well depending on your hours. If you have reports scheduled that make use of that data in xi or NNA it could be doing this as well.

[snapon_admin]

I forget the command, but isn't there a command I can run that will tell me all the scheduled reports that exist on the server?

[cmerchant]

It's run on the cron job by apache. To see if you have any scheduled reports, you can view the apache crontab:

Code: Select all
crontab -u apache -l

That was from a forum post: http://support.nagios.com/forum/viewtop ... 16&t=28472