Page 1 of 2

ESX3 CRITICAL - HOST IO Unknown error

Posted: Wed Nov 16, 2016 12:47 pm
by hondain
Hi
I used have issue with the vmware plugin, was giving "Unknown error" alert very often, and for no reason for all resources of esx. I fixed it with replacing the check_esx.pl with new updated script check_esx.pl which was Modified 10/12/2015 by Scott Wilkerson.The issue was fixed. But I update my Nagios XI last weekend from Nagios XI 5.2 to 5.3.2. Now its happening again. I saw the script its the same new script, still giving errors. Want to to know how to fix it ? Would really appreciate some help here ? attaching the script as well.

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Wed Nov 16, 2016 3:56 pm
by rkennedy
Does this happen all the time, or is it pretty 'random'? It sounds like it's random, which would make me think some sort of timeout or connection issue.

Also, how many services do you have running this check?

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Thu Nov 17, 2016 8:10 am
by hondain
There are 7 services for esxi 6.0.2, of which 4 services (cpu usage, MEM, I/O and N/W) shows this error. We don't monitor vmware services of guest OS. It is 5 to 6 times through out the day, even on time when esx is not being used. I have attached sample screenshort.

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Thu Nov 17, 2016 10:34 am
by rkennedy
Could you paste the service definitions for the CPU usage, MEM, I/O, and Network services for us to look at?

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Thu Nov 17, 2016 4:03 pm
by hondain
[root@LXHMIMON01 services]# cat HMIESX03.cfg
###############################################################################
#
# Service configuration file
#
# Created by: Nagios Core Config Manager 2.5.2
# Date: 2016-09-16 05:43:55
# Version: Nagios 3.x config file
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name HMIESX03
service_description CPU Usage for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!CPU!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Datastore usage for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!VMFS!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Input / Output for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!IO!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Memory for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!MEM!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Networking for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!NET!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Services for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!SERVICE!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description VM Status for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!RUNTIME!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Thu Nov 17, 2016 5:38 pm
by tgriep
Can you login to the Nagios server, run the following commands and post the output?

Code: Select all

cd /usr/local/nagios/libexec
./check_esx3.pl -V
su nagios
./check_esx3.pl -V
./check_esx3.pl -H xxx.xxx.xxx.xxx -f /usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt -l CPU
Replace xxx.xxx.xxx.xxx with the IP address of your VCenter server.

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Fri Nov 18, 2016 6:39 am
by hondain
$ cd /usr/local/nagios/libexec
$ ./check_esx3.pl -V
check_esx3.pl 0.2.1
$ ./check_esx3.pl -H hmiesx03.hmin.am.honda.com -f /usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt -l CPU
ESX3 OK - cpu usage=25122.00 MHz (34.50%) | cpu_usagemhz=25122.00Mhz;; cpu_usage=34.50%;;

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Fri Nov 18, 2016 11:39 am
by tgriep
Lets increase the timeout for that command to see if this helps the issue.
Login to the XI GUI and go to the Core Config Manager > commands menu and find the following command
check_esx3_host

Edit the command line and change the following from

Code: Select all

$USER1$/check_esx3.pl -H "$HOSTADDRESS$" -f "$ARG1$" -l "$ARG2$" $ARG3$
to

Code: Select all

$USER1$/check_esx3.pl -H "$HOSTADDRESS$" -t 59 -f "$ARG1$" -l "$ARG2$" $ARG3$
Save the change and apply the config.
Post back is this helps out.

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Mon Nov 21, 2016 10:15 am
by hondain
Din't fix the issue. Still getting those alerts

Re: ESX3 CRITICAL - HOST IO Unknown error

Posted: Mon Nov 21, 2016 12:11 pm
by tgriep
Try replacing the check_esx3.pl plugin on your server with the one attached to this post and see if that fixes the issue for you.
First, make a backup of the old plugin by running this as root on the XI server.

Code: Select all

cd /usr/local/nagios/libexec
mv check_esx3.pl check_esx3.old.pl
Then upload the attached plugin to the /usr/local/nagios/libexec folder and setup the permissions for it by running the following.

Code: Select all

chown apache.nagios check_esx3.pl
chmod 775 check_esx3.pl
Try that and see if the intermittent alerts issue is resolved for you.