Page 1 of 3

I/O Wait more than 20%

Posted: Fri May 05, 2017 6:15 pm
by ericssonvietnam
I am able to see that the I/O for the server is more than 20% and it shows to be in waiting state.

Can you suggest attached is the SS for reference.

Re: I/O Wait more than 20%

Posted: Mon May 08, 2017 10:50 am
by tmcdonald
We'll need a bit more information:
  • Do you have a RAM disk enabled?
  • What sort of disks are in use?
  • How many hosts and services are you monitoring?

Re: I/O Wait more than 20%

Posted: Sat May 13, 2017 1:03 am
by ericssonvietnam
tmcdonald wrote:We'll need a bit more information:
  • Do you have a RAM disk enabled?
  • What sort of disks are in use?
  • How many hosts and services are you monitoring?
As i can see in the below two output i think there is no RAM disk enable on my server let me know if i can check it from some other command :
+++++++++++++++++++++++++++++++++++++++++
[root@nagiosxi ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 909G 457G 407G 53% /
tmpfs 16G 80K 16G 1% /dev/shm
/dev/sda2 477M 68M 384M 15% /boot
+++++++++++++++++++++++++++++++++++++++++
[root@nagiosxi ~]# free -m
total used free shared buffers cached
Mem: 31941 31417 523 1 1620 26431
-/+ buffers/cache: 3364 28576
Swap: 4095 253 3842
++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
i have around 127 host configured and there are 1500 services in total
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Not aware to what type of disk we are using let me know if ypu can help me out in order to identify the same.

Re: I/O Wait more than 20%

Posted: Mon May 15, 2017 10:55 am
by tgriep
High I/O Wait usually points to the system cannot write files / data to the hard drive fast enough so it has to wait until it can.
First thing to try is to increase the amount of memory for the server and see if that helps. Adding more memory lets the system cache the writes to disk and that may help out in the performance.
Without knowing if the system is a physical server or if it is a virtual server, we cannot guess what type of drives the system is running on.
Talk to the person who setup the system and they should be able to provide the details needed.

Re: I/O Wait more than 20%

Posted: Fri May 26, 2017 1:11 pm
by ericssonvietnam
tgriep wrote:High I/O Wait usually points to the system cannot write files / data to the hard drive fast enough so it has to wait until it can.
First thing to try is to increase the amount of memory for the server and see if that helps. Adding more memory lets the system cache the writes to disk and that may help out in the performance.
Without knowing if the system is a physical server or if it is a virtual server, we cannot guess what type of drives the system is running on.
Talk to the person who setup the system and they should be able to provide the details needed.
Yes it is a physical server with below configuration:
[nagios@nagiosxi logbase]$ free -g
total used free shared buffers cached
Mem: 31 30 1 0 1 23
-/+ buffers/cache: 5 26
Swap: 3 0 3

number of core and cpu = 4

getting below error in /var/log/messages

May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi rsyslogd-2177: imuxsock begins to drop messages from pid 45593 due to rate-limiting

Can you suggest on the above logs

Re: I/O Wait more than 20%

Posted: Fri May 26, 2017 1:25 pm
by tgriep
You can try the changes in this KB article to see if it helps with the ndo2db queue errors.
https://support.nagios.com/kb/article/n ... eeded.html

Re: I/O Wait more than 20%

Posted: Fri May 26, 2017 2:13 pm
by ericssonvietnam
tgriep wrote:You can try the changes in this KB article to see if it helps with the ndo2db queue errors.
https://support.nagios.com/kb/article/n ... eeded.html
I have checked and found below parameter exist
kernel.msgmnb = 131072000
kernel.msgmax = 131072000

But kernel.msgmni = 256000 is not present in the file /etc/sysctl.conf do i need to add it would it help out in someway.

Re: I/O Wait more than 20%

Posted: Fri May 26, 2017 2:15 pm
by tgriep
Yes, if that option is missing, add it and see if it helps out.

Re: I/O Wait more than 20%

Posted: Sun May 28, 2017 12:32 am
by ericssonvietnam
tgriep wrote:Yes, if that option is missing, add it and see if it helps out.
Still i haven't done any changes yet.

Can you check the below apache error logs is it because of this.

[Sun May 28 10:50:20 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/login.php? ... f&noauth=1
[Sun May 28 10:50:22 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/index.php?
[Sun May 28 10:56:14 2017] [error] [client 10.10.164.212] PHP Notice: Undefined index: flash_msg in /usr/local/nagiosxi/html/login.php on line 79
[Sun May 28 10:56:53 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/login.php? ... f&noauth=1
[Sun May 28 10:57:05 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/index.php?
[Sun May 28 12:14:36 2017] [error] [client 10.10.164.201] PHP Notice: Undefined index: flash_msg in /usr/local/nagiosxi/html/login.php on line 79

Re: I/O Wait more than 20%

Posted: Tue May 30, 2017 10:05 am
by tgriep
Those messages in the Apache logs are only PHP Notices and they can be ignored.
I know it is confusing that the line says error but they are truly only Notices.