I/O Wait more than 20%

[ericssonvietnam]

I am able to see that the I/O for the server is more than 20% and it shows to be in waiting state.

Can you suggest attached is the SS for reference.

[tmcdonald]

We'll need a bit more information:

• Do you have a RAM disk enabled?
• What sort of disks are in use?
• How many hosts and services are you monitoring?

[ericssonvietnam]

We'll need a bit more information:

• Do you have a RAM disk enabled?
• What sort of disks are in use?
• How many hosts and services are you monitoring?

As i can see in the below two output i think there is no RAM disk enable on my server let me know if i can check it from some other command :
+++++++++++++++++++++++++++++++++++++++++
[root@nagiosxi ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 909G 457G 407G 53% /
tmpfs 16G 80K 16G 1% /dev/shm
/dev/sda2 477M 68M 384M 15% /boot
+++++++++++++++++++++++++++++++++++++++++
[root@nagiosxi ~]# free -m
total used free shared buffers cached
Mem: 31941 31417 523 1 1620 26431
-/+ buffers/cache: 3364 28576
Swap: 4095 253 3842
++++++++++++++++++++++++++++++++++++++++++++++++++++++++=
i have around 127 host configured and there are 1500 services in total
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Not aware to what type of disk we are using let me know if ypu can help me out in order to identify the same.

[tgriep]

High I/O Wait usually points to the system cannot write files / data to the hard drive fast enough so it has to wait until it can.
First thing to try is to increase the amount of memory for the server and see if that helps. Adding more memory lets the system cache the writes to disk and that may help out in the performance.
Without knowing if the system is a physical server or if it is a virtual server, we cannot guess what type of drives the system is running on.
Talk to the person who setup the system and they should be able to provide the details needed.

[ericssonvietnam]

High I/O Wait usually points to the system cannot write files / data to the hard drive fast enough so it has to wait until it can.
First thing to try is to increase the amount of memory for the server and see if that helps. Adding more memory lets the system cache the writes to disk and that may help out in the performance.
Without knowing if the system is a physical server or if it is a virtual server, we cannot guess what type of drives the system is running on.
Talk to the person who setup the system and they should be able to provide the details needed.

Yes it is a physical server with below configuration:
[nagios@nagiosxi logbase]$ free -g
total used free shared buffers cached
Mem: 31 30 1 0 1 23
-/+ buffers/cache: 5 26
Swap: 3 0 3

number of core and cpu = 4

getting below error in /var/log/messages

May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi ndo2db: Error: queue recv error.
May 27 01:11:56 nagiosxi rsyslogd-2177: imuxsock begins to drop messages from pid 45593 due to rate-limiting

Can you suggest on the above logs

[tgriep]

You can try the changes in this KB article to see if it helps with the ndo2db queue errors.
https://support.nagios.com/kb/article/n ... eeded.html

[ericssonvietnam]

You can try the changes in this KB article to see if it helps with the ndo2db queue errors.
https://support.nagios.com/kb/article/n ... eeded.html

I have checked and found below parameter exist
kernel.msgmnb = 131072000
kernel.msgmax = 131072000

But kernel.msgmni = 256000 is not present in the file /etc/sysctl.conf do i need to add it would it help out in someway.

[tgriep]

Yes, if that option is missing, add it and see if it helps out.

[ericssonvietnam]

Yes, if that option is missing, add it and see if it helps out.

Still i haven't done any changes yet.

Can you check the below apache error logs is it because of this.

[Sun May 28 10:50:20 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/login.php? ... f&noauth=1
[Sun May 28 10:50:22 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/index.php?
[Sun May 28 10:56:14 2017] [error] [client 10.10.164.212] PHP Notice: Undefined index: flash_msg in /usr/local/nagiosxi/html/login.php on line 79
[Sun May 28 10:56:53 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/login.php? ... f&noauth=1
[Sun May 28 10:57:05 2017] [error] [client 10.10.164.212] PHP Notice: Undefined offset: 1000 in /usr/local/nagiosxi/html/includes/components/helpsystem/helpsystem.inc.php on line 252, referer: http://10.10.164.52/nagiosxi/index.php?
[Sun May 28 12:14:36 2017] [error] [client 10.10.164.201] PHP Notice: Undefined index: flash_msg in /usr/local/nagiosxi/html/login.php on line 79

[tgriep]

Those messages in the Apache logs are only PHP Notices and they can be ignored.
I know it is confusing that the line says error but they are truly only Notices.