ESX3 CRITICAL - HOST IO Unknown error

[hondain]

Hi
I used have issue with the vmware plugin, was giving "Unknown error" alert very often, and for no reason for all resources of esx. I fixed it with replacing the check_esx.pl with new updated script check_esx.pl which was Modified 10/12/2015 by Scott Wilkerson.The issue was fixed. But I update my Nagios XI last weekend from Nagios XI 5.2 to 5.3.2. Now its happening again. I saw the script its the same new script, still giving errors. Want to to know how to fix it ? Would really appreciate some help here ? attaching the script as well.

[rkennedy]

Does this happen all the time, or is it pretty 'random'? It sounds like it's random, which would make me think some sort of timeout or connection issue.

Also, how many services do you have running this check?

[hondain]

There are 7 services for esxi 6.0.2, of which 4 services (cpu usage, MEM, I/O and N/W) shows this error. We don't monitor vmware services of guest OS. It is 5 to 6 times through out the day, even on time when esx is not being used. I have attached sample screenshort.

[rkennedy]

Could you paste the service definitions for the CPU usage, MEM, I/O, and Network services for us to look at?

[hondain]

[root@LXHMIMON01 services]# cat HMIESX03.cfg
###############################################################################
#
# Service configuration file
#
# Created by: Nagios Core Config Manager 2.5.2
# Date: 2016-09-16 05:43:55
# Version: Nagios 3.x config file
#
# --- DO NOT EDIT THIS FILE BY HAND ---
# Nagios CCM will overwrite all manual settings during the next update if you
# would like to edit files manually, place them in the 'static' directory or
# import your configs into the CCM by placing them in the 'import' directory.
#
###############################################################################

define service {
host_name HMIESX03
service_description CPU Usage for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!CPU!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Datastore usage for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!VMFS!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Input / Output for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!IO!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Memory for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!MEM!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Networking for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!NET!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description Services for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!SERVICE!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

define service {
host_name HMIESX03
service_description VM Status for VMHost
use xiwizard_generic_service
check_command check_esx3_host!/usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt!RUNTIME!
max_check_attempts 1
check_interval 5
retry_interval 1
check_period xi_timeperiod_24x7
notification_interval 60
notification_period xi_timeperiod_24x7
contacts vnc72000
_xiwizard vmware
register 1
}

###############################################################################
#
# Service configuration file
#
# END OF FILE
#
###############################################################################

[tgriep]

Can you login to the Nagios server, run the following commands and post the output?

Code: Select all

cd /usr/local/nagios/libexec
./check_esx3.pl -V
su nagios
./check_esx3.pl -V
./check_esx3.pl -H xxx.xxx.xxx.xxx -f /usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt -l CPU

Replace xxx.xxx.xxx.xxx with the IP address of your VCenter server.

[hondain]

$ cd /usr/local/nagios/libexec
$ ./check_esx3.pl -V
check_esx3.pl 0.2.1
$ ./check_esx3.pl -H hmiesx03.hmin.am.honda.com -f /usr/local/nagiosxi/etc/components/vmware/HMIESX03_auth.txt -l CPU
ESX3 OK - cpu usage=25122.00 MHz (34.50%) | cpu_usagemhz=25122.00Mhz;; cpu_usage=34.50%;;

[tgriep]

Lets increase the timeout for that command to see if this helps the issue.
Login to the XI GUI and go to the Core Config Manager > commands menu and find the following command
check_esx3_host

Edit the command line and change the following from

Code: Select all

$USER1$/check_esx3.pl -H "$HOSTADDRESS$" -f "$ARG1$" -l "$ARG2$" $ARG3$

to

Code: Select all

$USER1$/check_esx3.pl -H "$HOSTADDRESS$" -t 59 -f "$ARG1$" -l "$ARG2$" $ARG3$

Save the change and apply the config.
Post back is this helps out.

[hondain]

Din't fix the issue. Still getting those alerts

[tgriep]

Try replacing the check_esx3.pl plugin on your server with the one attached to this post and see if that fixes the issue for you.
First, make a backup of the old plugin by running this as root on the XI server.

Code: Select all

cd /usr/local/nagios/libexec
mv check_esx3.pl check_esx3.old.pl

Then upload the attached plugin to the /usr/local/nagios/libexec folder and setup the permissions for it by running the following.

Code: Select all

chown apache.nagios check_esx3.pl
chmod 775 check_esx3.pl

Try that and see if the intermittent alerts issue is resolved for you.