Page 1 of 2

I/O wait issue

Posted: Tue Jan 06, 2015 1:24 pm
by snapon_admin
Noticing a high percentage of I/O wait on our NNA server and wondering what options I have to reduce it. Our server has ~100-120 sources, 8 CPU cores (pretty sure it's 2 quad cores but it might be 4 dual cores), 8GB RAM, using the vSphere OVF template. This is what I'm looking at:
IO wait NNA.png
What do?

Re: I/O wait issue

Posted: Tue Jan 06, 2015 1:40 pm
by scottwilkerson
Does this machine share drives with any other machines (e.g. SAN or VM's)?

Does the server have spinning drives or SSD's?

Re: I/O wait issue

Posted: Tue Jan 06, 2015 4:46 pm
by snapon_admin
Ahh, good thought. I'll have to ask our VM admins as they own it, but I'd be willing to bet it's sharing drive space with other VMs and that it's likely not running on SSDs. I will check and get back to you. Not sure why I didn't think of the most obvious thing first....thanks!

Re: I/O wait issue

Posted: Tue Jan 06, 2015 5:45 pm
by cmerchant
Let us know what you find out. Thanks.

Re: I/O wait issue

Posted: Wed Jan 07, 2015 10:53 am
by snapon_admin
Looks like, as I suspected, our VMs are all on shared SAN storage. I don't see it being possible for that to be changed, so is there anything I can do to bring the I/O wait down at all?

Re: I/O wait issue

Posted: Wed Jan 07, 2015 5:16 pm
by sreinhardt
Are you able to correlate those spikes to anything in particular? Maybe when reports are being run, heavy user usage, maybe spikes in network traffic being pumped to NNA? The seem to be for relatively short (5-15 minute) windows and only when they overlap is it really hard on your system for longer periods of time.

Re: I/O wait issue

Posted: Thu Jan 08, 2015 10:54 am
by snapon_admin
Well I disabled the Network analyzer tab in XI, the one that shows up when you click on a host or service, and ever since the I/O spikes have almost completely disappeared. I'm assuming people were click on it, possibly not knowing what it even did, and since that tab runs a query against ALL sources on the NNA server it was causing the issue. Ever since I disabled the tab, there has still been a load spike at almost the same time both days. At around 8-8:30 am yesterday and the day before (after I disabled the tab) the I/O wait shoots up to about 30ish for an hour. After that all is well.

Re: I/O wait issue

Posted: Thu Jan 08, 2015 2:09 pm
by sreinhardt
That makes a lot of sense as to why the increased load. Seems like around the time people start coming into the office it spikes some, well depending on your hours. If you have reports scheduled that make use of that data in xi or NNA it could be doing this as well.

Re: I/O wait issue

Posted: Thu Jan 08, 2015 3:32 pm
by snapon_admin
I forget the command, but isn't there a command I can run that will tell me all the scheduled reports that exist on the server?

Re: I/O wait issue

Posted: Thu Jan 08, 2015 3:38 pm
by cmerchant
It's run on the cron job by apache. To see if you have any scheduled reports, you can view the apache crontab:

Code: Select all
crontab -u apache -l
That was from a forum post: http://support.nagios.com/forum/viewtop ... 16&t=28472