ESXI Plugin plugin spawns in the 1000s

[khouryi]

Nagiosxi 5.5.4
Redhat Linux 7.4
load average: 294.89, 242.34, 165.81
check_esx3 plugin spawns 1000s
Linux server crashes and reboots . Disabling the plugin for now. Any fix to this issue?

[npolovenko]

Hello, @khouryi. Are all the other plugins spawning fast or taking long time as well? Can you show me the command you're running against it?
Can you check the version of the plugin?

./check_esx3.pl -V

[khouryi]

# ./check_esx3.pl -V
check_esx3.pl 0.7.1
[root@sl79nag01 libexec]#

I had to disable the notification to most of them.
I used ps -ef |grep check_esx3.pl |wc -l and when the server was about to crash, it had 1001 running.

[npolovenko]

@khouryi, For some reason, the plugin did not time out and left hanging. Have you modified any service timeout settings in the nagios.cfg file or in the commands themselves? How many services are using check_esx3 plugin on your system? What are the check interval, max_check_attempts and retry interval values?

[khouryi]

what does this mean? we disabled all Vcenter alerts and still see this error below. I also confirmed the file does exist in Nagios.
Confirmation:
[root@sl79nag01 libexec]# ls -la check_esx
check_esx3.pl check_esx3.pl_backup check_esxi_hardware.py
[root@sl79nag01 libexec]# ls -la check_esx3.pl
-rwxrwxr-x 1 apache nagios 169540 Jul 18 16:44 check_esx3.pl
[root@sl79nag01 libexec]# pwd
/usr/local/nagios/libexec
[root@sl79nag01 libexec]#
[root@sl79nag01 libexec]# cksum check_esx3.pl
2725099542 169540 check_esx3.pl
[root@sl79nag01 libexec]#

Alert
***** Nagios XI Alert *****

Nagios has detected a problem with this service.

Notification Type: PROBLEM

Service: USBBBSP0001 Input / Output
Host: usbbvc1001
Address: usbbvc1001.corp.inbaxalta.com
State: CRITICAL
Info:
(No output on stdout) stderr: execvp(/usr/local/nagios/libexec/check_esx3.pl, ...) failed. errno is 2: No such file or directory
Date/Time: 2018-10-03 12:54:29

[tgriep]

Can you login to the Nagios server as root, run the following commands and post the output?

Code: Select all

su - nagios
/usr/local/nagios/libexec/check_esx3.pl -V

The plugin may exist, but if the Perl Modules it needs to run could be missing or that the permissions are set wrong and it is generating the error.

Then, open up the /usr/local/nagios/var/objects.cache file with an editor, find that service check and copy the configuration from the file and add it to the forum post.

Thanks

[khouryi]

[root@sl79nag01 var]# su - nagios
Last login: Tue Oct 9 22:00:07 EDT 2018
[nagios@sl79nag01 ~]$ /usr/local/nagios/libexec/check_esx3.pl -V
check_esx3.pl 0.7.1
[nagios@sl79nag01 ~]$

define command {
command_name check_esx3_guest
command_line $USER1$/check_esx3.pl -H "$HOSTADDRESS$" -f "$ARG1$" -N "$ARG2$" -l "$ARG3$" $ARG4$
}

define command {
command_name check_esx3_host
command_line $USER1$/check_esx3.pl -H "$HOSTADDRESS$" -f "$ARG1$" -l "$ARG2$" $ARG3$
}

define command {
command_name check_esx3_hosts
command_line $USER1$/check_esx3.pl -H "$HOSTADDRESS$" -u "$_HOSTUSER$" -p "$_HOSTPWD$" -l "$ARG1$" -s "$ARG2$" -w "$ARG3$" -c "$ARG4$"
}

define command {
command_name check_esx_hosts
command_line $USER1$/check_esx3.pl -H "$HOSTADDRESS$" -f "$ARG1$" -l "$ARG2$" -s "$ARG3$" -w "$ARG4$" -c "$ARG5$"
}

define command {
command_name check_esxi_hardware
command_line $USER1$/check_esxi_hardware.py -H $HOSTADDRESS$ -U $ARG1$ -P $ARG2$
}

[tgriep]

The -f option for the plugin is an authfile option that the plugin uses to get the username and password to login to the VMWare infrastructure.
Make sure your "$ARG1$" is pointing to a valid authfile and that the nagios user can access it.

[khouryi]

I can see the authfiles are accessible by Nagios, please see below
[nagios@sl79nag01 vmware]$ ls
sa79esx1001_corp_shire_com_auth.txt sa79esx1003_corp_shire_com_auth.txt sa79esx1021_auth.txt sy21esx20a_corp_shire_com_auth.txt usbbvc1001_corp_inbaxalta_com_auth.txt
sa79esx1002_corp_shire_com_auth.txt sa79esx1008_corp_shire_com_auth.txt sa79esx1021_corp_inbaxalta_com_auth.txt usbbesxe1001_corp_inbaxalta_com_auth.txt
[nagios@sl79nag01 vmware]$ ls -la
total 40
drwsrwsr-x 2 apache nagios 4096 Sep 25 07:47 .
drwsrwsr-x 4 apache nagios 97 Oct 6 2017 ..
-rw-r--r-- 1 apache nagios 35 Oct 6 2017 sa79esx1001_corp_shire_com_auth.txt
-rw-r--r-- 1 apache nagios 35 Oct 6 2017 sa79esx1002_corp_shire_com_auth.txt
-rw-r--r-- 1 apache nagios 35 Oct 6 2017 sa79esx1003_corp_shire_com_auth.txt
-rw-r--r-- 1 apache nagios 35 Oct 6 2017 sa79esx1008_corp_shire_com_auth.txt
-rw-r--r-- 1 apache nagios 35 Sep 25 07:47 sa79esx1021_auth.txt
-rw-r--r-- 1 apache nagios 35 Sep 20 08:50 sa79esx1021_corp_inbaxalta_com_auth.txt
-rw-r--r-- 1 apache nagios 39 Jul 24 09:48 sy21esx20a_corp_shire_com_auth.txt
-rw-r--r-- 1 apache nagios 55 Dec 7 2017 usbbesxe1001_corp_inbaxalta_com_auth.txt
-rw-r--r-- 1 apache nagios 55 Dec 7 2017 usbbvc1001_corp_inbaxalta_com_auth.txt
[nagios@sl79nag01 vmware]$ pwd
/usr/local/nagiosxi/etc/components/vmware
[nagios@sl79nag01 vmware]$ whoami
nagios
[nagios@sl79nag01 vmware]$

[tgriep]

Can you get the configuration settings for the "USBBBSP0001 Input / Output" service and post it here?

To do that, go to the Core Config Manager > Services menu and search for that service.
Click on the Floppy Icon in the right side of the menu which will bring up the configurations.
Copy that information and post it here so we can view it.