CHECK_VMWARE_API.PL CRITICAL - Host "" does not exist

[brennanma]

Hi Folks,

We are currently running a Nagios XI appliance (version 5.8.2). We are trying to monitor several VMWare ESXi hosts (version 6.7.0, 14320388). I have four identical hosts which are HP DL 380 Gen 9 -- we are running the HP customized version of ESXi.

Of the four hosts, three of them test without issue. The fourth, however, returns the error mentioned in the subject (CHECK_VMWARE_API.PL CRITICAL - Host "" does not exist). As far as I can tell, the hosts are configured identically. The checks are also working for 4 other VMWare ESXi hosts which are running different hardware but the same ESXi version.

This is actually a continuation of https://support.nagios.com/forum/viewto ... =6&t=56682. Shortly after creating the ticket, I lost access to the NagiosXI forum and therefore couldn't update my own post to respond to @lmiltchev. With the pandemic starting, I eventually put this task aside and then forgot about it. The problem is still occurring for this one host, and a Google search brought me back to my own post.

I got to the point where I was convinced the host must have had something bad in it's configuration. I migrated all the VM's off of it and re-installed ESXi from scratch. The problem persisted to the fresh install. I am now even more baffled than I was back in December 2019.

To respond to the questions from the previous post ...

Did you follow the instructions in the document below?
https://assets.nagios.com/downloads/nag ... ios-XI.pdf

Yes, I did. And it's working for multiple other hosts without issue.

Can you show us a command, run from the command line along with the output of it in "verbose mode"?

Code: Select all

/usr/local/nagios/libexec/check_vmware_api.pl -H "172.27.0.21" -u 'nagios' -p '<censored>' -l "CPU" -vvv
[extra-opts] check_vmware_api.pl -H 172.27.0.21 -u nagios -p <censored> -l CPU -vvv
Use of uninitialized value in concatenation (.) or string at /usr/local/nagios/libexec/check_vmware_api.pl line 963.
CHECK_VMWARE_API.PL CRITICAL - Host "" does not exist

Also, try the following:

Code: Select all

cd /usr/lib/vmware-vcli/apps/general
./connect.pl --url https://<ip address>/sdk/vimService --username '<username>' --password '<password>'

Were you able to connect?

Yes

Code: Select all

./connect.pl --url https://172.27.0.21/sdk/vimService --username 'nagios' --password '<censored>'

Connection Successful
Server Time : 2021-03-27T22:04:38.994158Z

[benjaminsmith]

HI,

Do you remember which version of the Vmware Perl SDK was installed on the XI server. It's not able to retrieve the host information.

A few more questions?

1. Did you use our wizard to set up these checks?
2. Do any of the checks return valid results (i.e. memory, networking, services, VM status..etc.)? If not, are you getting the same error message output for all of them?

It's failing here in the plugin.

Code: Select all

my $host_view = Vim::find_entity_views(view_type => 'HostSystem', filter => $host_name, properties => (defined($timeshift) ? [ 'name', 'runtime.inMaintenanceMode', 'configManager.dateTimeSystem' ] : [ 'name', 'runtime.inMaintenanceMode']) ); # Added properties named argument.
	die "Runtime error\n" if (!defined($host_view));
	die "Host \"" . $$host_name{"name"} . "\" does not exist\n" if (!@$host_view);

It cannot search the inventory using the API and find this host.

Best Regards,
Benjamin

[brennanma]

Hi Benjamin,

Do you remember which version of the Vmware Perl SDK was installed on the XI server. It's not able to retrieve the host information.

Now, it's running on 7.0U2. It was on 6.7U3 when I created the original post. I updated it to the most recent (7.0U2) before creating the new post.

Did you use our wizard to set up these checks?

I set them up myself, manually, before. However, I just tried to re-create them using the wizard and I get the same result. I have tried it both with the host IP and the FQDN just for kicks.

Do any of the checks return valid results (i.e. memory, networking, services, VM status..etc.)? If not, are you getting the same error message output for all of them?

On this host, negative. We're monitoring CPU Usage, Datastore Usage, I/O, Memory, Networking, Services, and VM Status. All return the exact same error message. On the other 7 hosts, all the checks work without issue.

I suppose there's a chance a made an error setting up the host originally, and replicated that exact same error when I rebuilt it. If so, though, I'm confused what that error was and how I failed to make it on the other 7 ESXi instances we're running. We have brought our vCenter instance up to 7.0 now. My next troubleshooting step, if you don't have any suggestions, will be to bring the cluster to ESXi 7.0U2 and see if that helps. It'll take me a bit to schedule that, though.

[mbellerue]

I think we're going to have to delve into the perl debugger to see what's going on.

First, we'll launch the debugger like this.

Code: Select all

perl -d /usr/local/nagios/libexec/check_vmware_api.pl -H "172.27.0.21" -u 'nagios' -p '<censored>' -l "CPU" -vvv

Next, we're going to add a couple of break points. You should be at a prompt that looks like __DB<1>__. Here are the commands we enter to set the break points

Code: Select all

b 505
b 614
b 2927
b 3000

You will use c to continue the script to the next break point.
So hit c and at the first break point we want to check the $host variable. Use x $host and see what the result is, then move to the second break point.

At the second break point, we want to see the $host_address variable, so x $host_address, see what the result is, and move to the next break point.

At the third break point, we want to see x $vmname, and move on.

At the fourth break point, we want to see x $values. At which point you can hit q to quit the debugger.

I'm not certain that it's going to hit the 3rd, and subsequently 4th break points. If it doesn't, we need to go down a different path to hit different points in the script. But if you could run this against a working host and the non-working host so that we can compare the values, that would be amazing.

[brennanma]

Thanks for the detailed instructions -- I'm still a novice with Perl. I'm thinking we might be running different versions (I have version 0.7.1 installed) of the check_vmware_api.pl script as the middle two line numbers provided (614 and 2927) weren't breakable. Looking at the code, I pushed those to 619 and 2933 to get the desired output. Unfortunately, it crashed before we hit break point 3.

Code: Select all

[root@nagios-ma ~]# perl -d /usr/local/nagios/libexec/check_vmware_api.pl -H "172.27.0.21" -u 'nagios' -p '<censored>' -l "CPU" -vvv

Loading DB routines from perl5db.pl version 1.37
Editor support available.

Enter h or 'h h' for help, or 'man perldebug' for more help.

CheckVMwareAPI::(/usr/local/nagios/libexec/check_vmware_api.pl:45):
45:     my $perl_module_instructions="
46:     Download the latest version of the vSphere SDK for Perl from VMware.
47:     In this example we use VMware-vSphere-Perl-SDK-5.1.0-780721.x86_64.tar.gz,
48:     but the instructions should apply to other versions as well.
  DB<1> b 505
  DB<2> b 619
  DB<3> b 2933
  DB<4> b 3000
  DB<5> c
[extra-opts] check_vmware_api.pl -H 172.27.0.21 -u nagios -p <censored> -l CPU -vvv
CheckVMwareAPI::main(/usr/local/nagios/libexec/check_vmware_api.pl:505):
505:            my $password = $np->opts->password;
  DB<5> x $host
0  '172.27.0.21'
  DB<6> c
CheckVMwareAPI::main(/usr/local/nagios/libexec/check_vmware_api.pl:619):
619:                    $host_address .= ":443" if (index($host_address, ":") == -1);
  DB<6> x $host_address
0  '172.27.0.21'
  DB<7> c
Use of uninitialized value in concatenation (.) or string at /usr/local/nagios/libexec/check_vmware_api.pl line 963.
 at /usr/local/nagios/libexec/check_vmware_api.pl line 963.
        CheckVMwareAPI::return_host_performance_values(undef, 'HASH(0x3244a20)', 'cpu', 'usagemhz.average', 'usage.average') called at /usr/local/nagios/libexec/check_vmware_api.pl line 1361
        CheckVMwareAPI::host_cpu_info(undef, 'Nagios::Monitoring::Plugin=HASH(0x3236890)', undef, undef) called at /usr/local/nagios/libexec/check_vmware_api.pl line 687
        eval {...} called at /usr/local/nagios/libexec/check_vmware_api.pl line 585
        CheckVMwareAPI::main() called at /usr/local/nagios/libexec/check_vmware_api.pl line 832
CHECK_VMWARE_API.PL CRITICAL - Host "" does not exist
Debugged program terminated.  Use q to quit or R to restart,
  use o inhibit_exit to avoid stopping after program termination,
  h q, h R or h o to get additional info.  
  DB<7> q

That did give a bit more insight. It appears that $host is undefined when it calls return_host_performance_values. It's also null when host_cpu_info is called. It seems that $host comes from $esx (line 684 for me) but that is undef because $datacenter is undef. I tried defining the datacenter (even though this isn't required for the other hosts) just for the heck of it.
There's no longer an exception, but it still doesn't work:

Code: Select all

[root@nagios-ma ~]# /usr/local/nagios/libexec/check_vmware_api.pl -H "vmware-ma-3.contoso.com" -D "vmware-ma-3.contoso.com" -u 'nagios' -p '<censored>' -l "CPU" -vvv[extra-opts] check_vmware_api.pl -H vmware-ma-3.contoso.com -D vmware-ma-3.contoso.com -u nagios -p <censored> -l CPU -vvv
CHECK_VMWARE_API.PL CRITICAL - Host "vmware-ma-3.contoso.com" does not exist

(No, we don't really use contoso.com)

[mbellerue]

Excellent digging! That line 684 is part of a larger if statement. I mentioned that I wasn't sure that we'd hit break points 3 or 4, and that's because I thought for some reason it was hitting that elsif at line 681, which as you were able to confirm, it is. I suspect that if you run the same break points against an ESXi machine where this script is returning properly, it will hit the beginning part of the if statement at line 653, or the final else statement at line 759. If we can confirm that a working example goes down a different path, maybe we can figure out how to guide the non-working example down that same path. Or if the working example goes down the same path, we'll have to find some more break points and poke the script a little more.

Also, quick question. Is your VMWare datacenter name actually the same as the name of that particular ESXi machine? I suspect we'd still be hitting this issue if it was set to the correct datacenter name, because like I said, I think it's going down an incorrect path in the script. However, I just want to make sure that the test you ran is failing for the correct reasons, if that makes sense.

[mbellerue]

Oh I also forgot to call out, good catch on the version number thing. I was running the same version as you, but apparently I had modified my version a bit, and that's why the line numbers were off. I have refreshed with a clean copy of the script, and just to be sure, if we have to set up any more break points, I will mention the line numbers, and show what I expect the line to be.

[brennanma]

Strangely, everything seems to be undef with the working host as well. I am now sure that problem is the VMWare API call at line 961:

Code: Select all

my $host_view = Vim::find_entity_views(view_type => 'HostSystem', filter => $host_name, properties => (defined($timeshift) ? [ 'name', 'runtime.inMaintenanceMode', 'configManager.dateTimeSystem' ] : [ 'name', 'runtime.inMaintenanceMode']) ); # Added properties named argument.

When I break at 962 (both working and non-working) $host_name is undef. For some reason, on the working one Vim::find_entity_views still returns valid information (even though we're filtering against an undef variable):

Code: Select all

  DB<4> x $host_name
0  undef
  DB<5> x $host_view
0  ARRAY(0x1ea4230)
   0  HostSystem=HASH(0x351a700)
      'mo_ref' => ManagedObjectReference=HASH(0x34a5238)
         'type' => 'HostSystem'
         'value' => 'ha-host'
      'name' => 'vmware-ma-1.contoso.com'
      'runtime.inMaintenanceMode' => 'false'
      'vim' => Vim=HASH(0x17eb9b8)
      (removing the rest as it's a very large array)

On the problematic host, $host_view comes back as an empty array:

Code: Select all

  DB<2> x $host_name
0  undef
  DB<3> x $host_view
0  ARRAY(0x4fdccd8)
     empty array

Also, quick question. Is your VMWare datacenter name actually the same as the name of that particular ESXi machine?

Negative. I'm trying to query the host directly, not go through vCenter. In order to do that while specifying a datacenter name, the datacenter name needs to be the host name.

[mbellerue]

961 is the last thing to be run before the script dies. But is it being called from the same place? There are a lot of places that call the return_host_performance_values function. Let's try a new set of break points against the working and non-working hosts.

Code: Select all

b 1281
b 1303
b 1326
b 1361

Each of the above break points are an area where return_host_performance_values is called from the host_cpu_info function. Both working and non-working hosts should hit one of these. Ideally they would both hit the same one, and everything would be happy. Check which line the two hosts break on to see if they're the same.

If they're not the same, run x @_ at the line where they break. That should dump everything in the array that was passed to the host_cpu_info function. Hopefully from that information, we can tease out any differences between the data that the working host got and the data that the non-working host got.

Thank you very much for sticking with me through this troubleshooting!

[brennanma]

The working and non-working both break at the same lines -- 1281 and 1361

Working:

Code: Select all

  DB<1> b 1281
  DB<2> b 1303
  DB<3> b 1326
  DB<4> b 1361
  DB<5> c
[extra-opts] check_vmware_api.pl -H 172.27.0.5 -u nagios -p <redacted> -l CPU -vvv
CheckVMwareAPI::host_cpu_info(/usr/local/nagios/libexec/check_vmware_api.pl:1281):
1281:           my $res = CRITICAL;
  DB<5> c
CheckVMwareAPI::host_cpu_info(/usr/local/nagios/libexec/check_vmware_api.pl:1361):
1361:                           $values = return_host_performance_values($host, $defperfargs, 'cpu', ('usagemhz.average', 'usage.average'));
  DB<5> c
CHECK_VMWARE_API.PL OK - cpu usage=2995.00 MHz (5.43%) | cpu_usagemhz=2995.00;; cpu_usage=5.43%;;
Debugged program terminated.  Use q to quit or R to restart,

Non-working:

Code: Select all

  DB<1> b 1281
  DB<2> b 1303
  DB<3> b 1326
  DB<4> b 1361
  DB<5> c
[extra-opts] check_vmware_api.pl -H 172.27.0.21 -u nagios -p <redacted> -l CPU -vvv
CheckVMwareAPI::host_cpu_info(/usr/local/nagios/libexec/check_vmware_api.pl:1281):
1281:           my $res = CRITICAL;
  DB<5> c
CheckVMwareAPI::host_cpu_info(/usr/local/nagios/libexec/check_vmware_api.pl:1361):
1361:                           $values = return_host_performance_values($host, $defperfargs, 'cpu', ('usagemhz.average', 'usage.average'));
  DB<5> c
Use of uninitialized value in concatenation (.) or string at /usr/local/nagios/libexec/check_vmware_api.pl line 963.
 at /usr/local/nagios/libexec/check_vmware_api.pl line 963.
        CheckVMwareAPI::return_host_performance_values(undef, 'HASH(0x19edf58)', 'cpu', 'usagemhz.average', 'usage.average') called at /usr/local/nagios/libexec/check_vmware_api.pl line 1361
        CheckVMwareAPI::host_cpu_info(undef, 'Nagios::Monitoring::Plugin=HASH(0x19e5890)', undef, undef) called at /usr/local/nagios/libexec/check_vmware_api.pl line 687
        eval {...} called at /usr/local/nagios/libexec/check_vmware_api.pl line 585
        CheckVMwareAPI::main() called at /usr/local/nagios/libexec/check_vmware_api.pl line 832
CHECK_VMWARE_API.PL CRITICAL - Host "" does not exist
Debugged program terminated.  Use q to quit or R to restart,

The weird part is that everything looks the same entering return_host_performance_values.

Working:

Code: Select all

  DB<1> b 957
  DB<2> c
[extra-opts] check_vmware_api.pl -H 172.27.0.5 -u nagios -p <redacted> -l CPU -vvv
CheckVMwareAPI::return_host_performance_values(/usr/local/nagios/libexec/check_vmware_api.pl:957):
957:            my $values;
  DB<2> x @_
0  undef
1  HASH(0x205fd18)
     empty hash
2  'cpu'
3  'usagemhz.average'
4  'usage.average'
  DB<3> q

Non-working:

Code: Select all

  DB<1> b 957
  DB<2> c
[extra-opts] check_vmware_api.pl -H 172.27.0.21 -u nagios -p <redacted> -l CPU -vvv
CheckVMwareAPI::return_host_performance_values(/usr/local/nagios/libexec/check_vmware_api.pl:957):
957:            my $values;
  DB<2> x @_
0  undef
1  HASH(0x1899d18)
     empty hash
2  'cpu'
3  'usagemhz.average'
4  'usage.average'
  DB<3> q