check_wmi_plus results differ from Nagios and command line

[hhlodge]

Nagios is reporting this for a check on a disk which started failing out of nowhere.

[1320265303] SERVICE ALERT: bt-w8k103;Drive E:;UNKNOWN;HARD;3;UNKNOWN - Could not find a drive matching 'E'. Available Drives are C:

Not Nagios's problem, right? So I go to debug from command iine, and it works just fine.

/usr/local/nagios/libexec/check_wmi_plus.pl -H bt-w8k103 -u xxxxx -p zzzzzzzz -m checkdrivesize -a E: -w 85 -c 95
OK - E: Total=40.00GB, Used=24.80GB (62.0%), Free=15.20GB (38.0%) |'E: Space'=24.80GB; 'E: Utilisation'=62.0%;85;95;

And this is consistently failing in Nagios and successful from command line. This makes troubleshooting tough. I restarted Nagios and the WMI service on the client to no avail.Any thoughts on how this could happen and what I might be able to do to see why Nagios is seeing what it is?

[mguthrie]

Can you show us the service definition for this check?

[hhlodge]

define service {
host_name bt-w8k103
service_description Drive E:
use xiwizard_windowswmi_service
check_command check_xi_service_wmiplus!checkdrivesize!-a E -w 85 -c 95!!!!!!
max_check_attempts 3
check_interval 5
retry_interval 1
first_notification_delay 0
notification_options w,u,r,c,f,s
notifications_enabled 1
contact_groups is,it-admins
_xiwizard windowswmi
register 1
}

[lmiltchev]

Kyle,

I would guess that you edited your config file and removed the username and the password from the service definition for security reasons, but in case you DON'T have this defined, try:

define service {
host_name bt-w8k103
service_description Drive E:
use xiwizard_windowswmi_service
check_command check_xi_service_wmiplus!<username>!<password>!checkdrivesize!-a 'E:' -w 85 -c 95!!!!!!
max_check_attempts 3
check_interval 5
retry_interval 1
first_notification_delay 0
notification_options w,u,r,c,f,s
notifications_enabled 1
contact_groups is,it-admins
_xiwizard windowswmi
register 1
}

[hhlodge]

No, I have the username and a password macro in the command itself, so I can change it in one place.

USER1$/check_wmi_plus.pl -H $HOSTADDRESS$ -u xxxx\\zzzzzzz -p $USER10$ -m $ARG1$ $ARG2$

Besides, if that was the issue, I'd get a NT_STATUS_ACCESS_DENIED failure.

[hhlodge]

I managed to get Nagios to output the debug mode output to a file by using redirects in the command and something is clearly different when Nagios runs it and from the command line. Here's the two outputs, command line being first. This is using a new version 1.481 from the plugin developer trying to assist me with this issue.

From command line:

Code: Select all

[root@psm-itmon libexec]# /usr/local/nagios/libexec/check_wmi_plus-1.48.pl -H bt-w8k103 -u xxxx -p zzzz -m checkvolsize -a E: -w 85 -c 95 -d
Command Line (v1.481): /usr/local/nagios/libexec/check_wmi_plus-1.48.pl -H bt-w8k103 -u USER -p PASS -m checkvolsize -a E: -w 85 -c 95 -d
Conf File Dir: /usr/local/packages/check_wmi_plus.v1.481
Loaded Conf File /usr/local/packages/check_wmi_plus.v1.481/check_wmi_plus.conf
Round #1 of 1
QUERY: /usr/local/bin/wmic --namespace root/cimv2 -U USER%PASS //bt-w8k103 'Select Capacity,DeviceID,DriveLetter,DriveType,FileSystem,FreeSpace,Label,Name from Win32_Volume where DriveType=3'
OUTPUT: CLASS: Win32_Volume
Capacity|DeviceID|DriveLetter|DriveType|FileSystem|FreeSpace|Label|Name
42947571712|\\?\Volume{1d5d15c1-3e8f-11e0-a934-806e6f6e6963}\|C:|3|NTFS|16081522688|(null)|C:\
42946523136|\\?\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\|E:|3|NTFS|16316686336|New Volume|E:\

COLUMNS:Capacity|DeviceID|DriveLetter|DriveType|FileSystem|FreeSpace|Label|Name
Now looking for (.*?)\n (use_split=1)
FIELDS:COLNAME=Capacity,FIELD=42947571712
COLNAME=DeviceID,FIELD=\\?\Volume{1d5d15c1-3e8f-11e0-a934-806e6f6e6963}\
COLNAME=DriveLetter,FIELD=C:
COLNAME=DriveType,FIELD=3
COLNAME=FileSystem,FIELD=NTFS
COLNAME=FreeSpace,FIELD=16081522688
COLNAME=Label,FIELD=(null)
COLNAME=Name,FIELD=C:\

Row Data Found OK
FIELDS:COLNAME=Capacity,FIELD=42946523136
COLNAME=DeviceID,FIELD=\\?\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\
COLNAME=DriveLetter,FIELD=E:
COLNAME=DriveType,FIELD=3
COLNAME=FileSystem,FIELD=NTFS
COLNAME=FreeSpace,FIELD=16316686336
COLNAME=Label,FIELD=New Volume
COLNAME=Name,FIELD=E:\

Row Data Found OK
WMI DATA:$VAR1 = [
          [
            {
              'DriveLetter' => 'C:',
              '_QuerySum_Capacity' => '42947571712',
              'FileSystem' => 'NTFS',
              '_ColSum_Capacity' => '85894094848',
              'Capacity' => '42947571712',
              '_ColSum_FreeSpace' => '32398209024',
              'Label' => '(null)',
              '_QuerySum_FreeSpace' => '16081522688',
              '_ChecksOK' => 1,
              'DeviceID' => '\\\\?\\Volume{1d5d15c1-3e8f-11e0-a934-806e6f6e6963}\\',
              'DriveType' => '3',
              'FreeSpace' => '16081522688',
              '_ItemCount' => 2,
              'Name' => 'C:\\'
            },
            {
              'DriveLetter' => 'E:',
              '_QuerySum_Capacity' => '42946523136',
              'FileSystem' => 'NTFS',
              'Capacity' => '42946523136',
              '_QuerySum_FreeSpace' => '16316686336',
              'Label' => 'New Volume',
              'DeviceID' => '\\\\?\\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\\',
              'DriveType' => '3',
              'FreeSpace' => '16316686336',
              'Name' => 'E:\\'
            }
          ]
        ];
Testing TEST VALUES $VAR1 = {
          'Capacity' => '42946523136',
          '_QuerySum_FreeSpace' => '16316686336',
          'Label' => 'New Volume',
          '_UsedSpace' => '26629836800',
          '_Free%' => '38.0',
          'FreeSpace' => '16316686336',
          '_DriveSizeGB' => '40.00',
          'DriveLetter' => 'E:',
          '_QuerySum_Capacity' => '42946523136',
          'FileSystem' => 'NTFS',
          '_Used%' => '62.0',
          'DeviceID' => '\\\\?\\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\\',
          '_FreeGB' => '15.20',
          'DriveType' => '3',
          'Name' => 'E:\\',
          '_UsedGB' => '24.80'
        };
WARNING SPECS: $VAR1 = [
          '85'
        ];
CRITICAL SPECS: $VAR1 = [
          '95'
        ];
------------ Critical Check ------------
Testing SPEC: 95
SPEC=_Used%,,95,
Range Spec - FIELD=_Used%, AT= MIN=0 MINMULTIPLIER= MAX=95 MAXMULTIPLIER=
Testing MIN: '0' for Field _Used% which has value: 62.0
TEST2 62.0 < 0
Test of <0 (0) vs 62.0 yields 0
Testing MAX: '95' for Field _Used% which has value: 62.0
TEST4 62.0 > 95
Test of 95 (95) vs 62.0 yields 0
Test Result = 0, Perf Spec=95, Trigger Display=, Field Tested=_Used%
------------ Warning Check ------------
Testing SPEC: 85
SPEC=_Used%,,85,
Range Spec - FIELD=_Used%, AT= MIN=0 MINMULTIPLIER= MAX=85 MAXMULTIPLIER=
Testing MIN: '0' for Field _Used% which has value: 62.0
TEST2 62.0 < 0
Test of <0 (0) vs 62.0 yields 0
Testing MAX: '85' for Field _Used% which has value: 62.0
TEST4 62.0 > 85
Test of 85 (85) vs 62.0 yields 0
Test Result = 0, Perf Spec=85, Trigger Display=, Field Tested=_Used%
------------ End Check ------------
Test Results:
Warn Perf Specs=$VAR1 = {
          '_FreeGB' => '',
          '_Free%' => '',
          '_Used%' => '85',
          '_UsedGB' => ''
        };
Warn Results=$VAR1 = [
          ''
        ];
Critical Perf Spec=$VAR1 = {
          '_FreeGB' => '',
          '_Free%' => '',
          '_Used%' => '95',
          '_UsedGB' => ''
        };
Critical Results=$VAR1 = [
          ''
        ];
Data Passed back from check: $VAR1 = {
          '_TestResult' => 0,
          'Capacity' => '42946523136',
          '_QuerySum_FreeSpace' => '16316686336',
          'Label' => 'New Volume',
          '_UsedSpace' => '26629836800',
          '_Free%' => '38.0',
          'FreeSpace' => '16316686336',
          '_DriveSizeGB' => '40.00',
          'DriveLetter' => 'E:',
          '_QuerySum_Capacity' => '42946523136',
          'FileSystem' => 'NTFS',
          '_Used%' => '62.0',
          '_DisplayMsg' => 'OK',
          '_StatusType' => 'OK',
          'DeviceID' => '\\\\?\\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\\',
          '_FreeGB' => '15.20',
          'DriveType' => '3',
          'Name' => 'E:\\',
          '_UsedGB' => '24.80',
          '_Triggers' => ''
        };
---------- Building Up Display
Incoming Data $VAR1 = {
          '_TestResult' => 0,
          'Capacity' => '42946523136',
          '_arg5' => '',
          '_UsedSpace' => '26629836800',
          '_nodatamode' => '',
          '_Free%' => '38.0',
          'FreeSpace' => '16316686336',
          '_DriveSizeGB' => '40.00',
          'FileSystem' => 'NTFS',
          '_arg1' => 'E:',
          '_arg2' => '',
          'VolumeDisplayName' => 'E:\\',
          '_bytefactor' => '',
          '_UsedGB' => '24.80',
          '_nodataexit' => '',
          '_nodatastring' => 'WMI Query returned no data. The item you were looking for may NOT exist or the software that creates the WMI Class may not be running, or all data has been excluded.
',
          'Label' => 'New Volume',
          '_QuerySum_FreeSpace' => '16316686336',
          '_host' => 'bt-w8k103',
          'DriveLetter' => 'E:',
          '_arg3' => '',
          '_QuerySum_Capacity' => '42946523136',
          '_Used%' => '62.0',
          '_DisplayMsg' => 'OK',
          '_timeout' => '',
          '_FreeGB' => '15.20',
          'DeviceID' => '\\\\?\\Volume{256a84e0-3e92-11e0-88c1-005056855d2b}\\',
          '_StatusType' => 'OK',
          '_delay' => '',
          'DriveType' => '3',
          'Name' => 'E:\\',
          '_arg4' => undef,
          '_Triggers' => ''
        };
------- Processing _DisplayMsg||~|~| - ||
Complex Format:_DisplayMsg,,~,~, - ,,
_DisplayMsg||~|~| - || ----> OK - 
------- Processing VolumeDisplayName||~|~| ||
Complex Format:VolumeDisplayName,,~,~, ,,
VolumeDisplayName||~|~| || ----> E:\ 
------- Processing _DriveSizeGB|GB|Total||||
Complex Format:_DriveSizeGB,GB,Total,,,,
_DriveSizeGB|GB|Total|||| ----> Total=40.00GB, 
------- Processing _UsedGB|GB|Used|| ||
Complex Format:_UsedGB,GB,Used,, ,,
_UsedGB|GB|Used|| || ----> Used=24.80GB 
------- Processing _Used%|%|~|~||(|)
Complex Format:_Used%,%,~,~,,(,)
_Used%|%|~|~||(|) ----> (62.0%), 
------- Processing _FreeGB|GB|Free|| ||
Complex Format:_FreeGB,GB,Free,, ,,
_FreeGB|GB|Free|| || ----> Free=15.20GB 
------- Processing _Free%|%|~|~||(|)
Complex Format:_Free%,%,~,~,,(,)
_Free%|%|~|~||(|) ----> (38.0%), 
---------- Building Up Performance Data
------- Processing _UsedGB|GB|{VolumeDisplayName} Space
Complex Format:_UsedGB,GB,{VolumeDisplayName} Space
_UsedGB|GB|{VolumeDisplayName} Space (Field=_UsedGB) ----> 'E:\ Space'=24.80GB;;; 
------- Processing _Used%|%|{VolumeDisplayName} Utilisation
Complex Format:_Used%,%,{VolumeDisplayName} Utilisation
_Used%|%|{VolumeDisplayName} Utilisation (Field=_Used%) ----> 'E:\ Utilisation'=62.0%;85;95; 
---------- Done
OK - E:\ Total=40.00GB, Used=24.80GB (62.0%), Free=15.20GB (38.0%)     |'E:\ Space'=24.80GB; 'E:\ Utilisation'=62.0%;85;95; 

From Nagios:

Code: Select all

# cat /tmp/out
Command Line (v1.481): /usr/local/nagios/libexec/check_wmi_plus-1.48.pl -H 10.70.19.67 -u USER -p PASS -m checkvolsize -a E: -w 85 -c 95 -d
Conf File Dir: /usr/local/packages/check_wmi_plus.v1.481
Loaded Conf File /usr/local/packages/check_wmi_plus.v1.481/check_wmi_plus.conf
Round #1 of 1
QUERY: /usr/local/bin/wmic --namespace root/cimv2 -U USER%PASS //10.70.19.67 'Select Capacity,DeviceID,DriveLetter,DriveType,FileSystem,FreeSpace,Label,Name from Win32_Volume where DriveType=3'
OUTPUT: CLASS: Win32_Volume
Capacity|DeviceID|DriveLetter|DriveType|FileSystem|FreeSpace|Label|Name
32210128896|\\?\Volume{84270cad-1a68-11e0-b457-806e6f6e6963}\|C:|3|NTFS|3284213760|(null)|C:\

COLUMNS:Capacity|DeviceID|DriveLetter|DriveType|FileSystem|FreeSpace|Label|Name
Now looking for (.*?)\n (use_split=1)
FIELDS:COLNAME=Capacity,FIELD=32210128896
COLNAME=DeviceID,FIELD=\\?\Volume{84270cad-1a68-11e0-b457-806e6f6e6963}\
COLNAME=DriveLetter,FIELD=C:
COLNAME=DriveType,FIELD=3
COLNAME=FileSystem,FIELD=NTFS
COLNAME=FreeSpace,FIELD=3284213760
COLNAME=Label,FIELD=(null)
COLNAME=Name,FIELD=C:\

Row Data Found OK
WMI DATA:$VAR1 = [
          [
            {
              'DriveLetter' => 'C:',
              '_QuerySum_Capacity' => '32210128896',
              'FileSystem' => 'NTFS',
              '_ColSum_Capacity' => '32210128896',
              'Capacity' => '32210128896',
              '_ColSum_FreeSpace' => 3284213760,
              'Label' => '(null)',
              '_QuerySum_FreeSpace' => 3284213760,
              '_ChecksOK' => 1,
              'DeviceID' => '\\\\?\\Volume{84270cad-1a68-11e0-b457-806e6f6e6963}\\',
              'DriveType' => '3',
              'FreeSpace' => 3284213760,
              '_ItemCount' => 1,
              'Name' => 'C:\\'
            }
          ]
        ];
UNKNOWN - Could not find a volume matching 'E:'. Available Volumes are C:\

[hhlodge]

I see the issue now and it's embarassing. The host got a new IP address so I was querying a host that didn't have an E drive, so it was perfectly correct. A lesson in using IP addresses in Nagios in a DHCP environment. Argggg. Sorry for the wasted bandwidth.

[lmiltchev]

I am also glad you figured this out!

[hhlodge]

On an aside, one thing i dislike about using hostnames rather than IP addresses for DHCP clients is in notifications and the GUI, I see the HOSTNAME and HOSTADDRESS fields as identical. I'd love to be able to see IP addresses as resolved by Nagios, at least an $IPADDRESS$ macro for notifications. I suspect that's a change in Nagios Core, yes?

[mguthrie]

Yeah, it's a change in core, and it wouldn't be a small one. I've heard this brought up before among the Core development team, and my understanding is that the "address" field is intended to be generic, so that it can emcompass IP4,IP6, FQDN, a MAC address, or whatever is needed to be referenced by a check command. Nagios itself doesn't resolve any IP addresses. That happens with the individual plugins. Your could add a custom macro that show up in notifications, but if those addresses are changing constantly what would be challenging to maintain.