Auth file suddenly stopped working

[N4g10s4dm1n]

CentOS 7
Nagios Core version 4.4.1

Hello,

I'm using check wmi plus to monitor windows servers. I've had a successful set up (wmi checks working and email alerts working) for about a year with no real issues. I'm using a credentials file that is called in the wmi checks to provide the user account credentials for remote wmi queries. This has worked flawlessly since I first got it up and running about a year ago. Out of nowhere, two nights ago, the user account (domain user account) that is used began to lock out, over and over. All checks are failing with the following message: "The WMI query had problems. You might have your username/password wrong or the user's access level is too low. Your authentication file might be incorrectly formatted or inaccessible.

Here's what I've done so far to troubleshoot:

Made sure the account credentials I'm using are not expired, and have not been reset.
Successfully authenticated manually with the account credentials in question to some of the Windows servers via WMI using a powershell command.
created a new authentication file with the same credentials and edited all checks to use the new file.
Checked Windows firewall and ensured WMI rules are still in place.
Checked network firewall for any blocked traffic from Nagios server to Windows servers and found none.

Any ideas/help is greatly appreciated!

Thank you.

[mcapra]

Which version of the check_wmi_plus.pl plugin are you using?

Can you run the check_wmi_plus.pl plugin from the command-line with the debug flag enabled and share the output?

Code: Select all

-d  Enable debug. Use this to see a lot more of what is going on including the exact WMI Query and results. User/passwords should be masked in the resulting output unless -z is specified.

You can provide -z without sharing the output here to validate that the creds are being correctly loaded from the file -- I'll take your work for it

Run that same command as the Nagios daemon user (typically, nagios):

Code: Select all

su nagios
/path/to/check_wmi_plus.pl

Can you check the perms of your authentication file to make sure the Nagios daemon user (typically, nagios) can read+stat that credentials file?

[cdienger]

@N4g10s4dm1n: Let us know if the direction @mcapra's provided has helped and provide us with the requested data if further assistance is needed.

[N4g10s4dm1n]

Thank you for your reply mcapra.

check_wmi_plus.pl version is 1.6

Output from running -d -z (output didn't show any credentials even with -z specified).

Code: Select all

[root@myserver ~]# /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -A /usr/local/nagios/etc/auth.txt
Command Line (v1.6): /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -A /usr/local/nagios/etc/auth.txt
Base Dir: /usr/local/nagios/libexec
Conf File Dir: /usr/local/nagios/libexec
Loaded Conf File /usr/local/nagios/libexec/check_wmi_plus.conf
Starting Keep State Mode
STATE FILE: /tmp/cwpss_checkcpu__myserver___.state
Round #1 of 1
QUERY: /usr/bin/wmic '-A' '/usr/local/nagios/etc/auth.txt' '--namespace' 'root/cimv2' '//myserver.mydomain' 'select PercentProcessorTime,Timestamp_Sys100NS from Win32_PerfRawData_PerfOS_Processor where Name="_Total"'
OUTPUT: [librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

Could not find the CLASS: line - an error occurred
WMI DATA:$VAR1 = undef;
UNKNOWN - The WMI query had problems. You might have your username/password wrong or the user's access level is too low.  Your Authentication File might be incorrectly formatted or inaccessible. Wmic error text on the next line.
[librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

I'm not sure what the Nagios daemon user is. When I set up Nagios I only created one user and that is the "nagiosadmin" user that is used to sign into the web interface.

Thank you for time and assistance. Very much appreciated!

[cdienger]

What does the username format look like in the file?

Other people that have run into this error were running into it because the username wasn't formatted correctly. Try running the command without the file and with the username and password instead:

/usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -u domainname/\username -p password

note the /\ that separates the domainname from the username.

[N4g10s4dm1n]

The format of the auth file, which was working for the past year and has not been changed, is like below:
username=accountname
password=thepasswordhere
domain=mydomain

Running the same check with specified username and password produces the same result.

Code: Select all

[root@myserver ~]# /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -u mydomain/\accountname -p password
Command Line (v1.6): /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -u mydomain/accountname -p password
Base Dir: /usr/local/nagios/libexec
Conf File Dir: /usr/local/nagios/libexec
Loaded Conf File /usr/local/nagios/libexec/check_wmi_plus.conf
Starting Keep State Mode
STATE FILE: /tmp/cwpss_checkcpu__myserver___.state
Round #1 of 1
QUERY: /usr/bin/wmic '-U' 'mydomain/accountname%password' '--namespace' 'root/cimv2' '//myserver.mydomain' 'select PercentProcessorTime,Timestamp_Sys100NS from Win32_PerfRawData_PerfOS_Processor where Name="_Total"'
OUTPUT: [librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

Could not find the CLASS: line - an error occurred
WMI DATA:$VAR1 = undef;
UNKNOWN - The WMI query had problems. You might have your username/password wrong or the user's access level is too low.  Your Authentication File might be incorrectly formatted or inaccessible. Wmic error text on the next line.
[librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

Side note, I was able to use WBEMTEST to successfully connect remotely to the root/cimv2 namesapce with the same account I am using in my auth file and in the commands above.

[cdienger]

Check out https://support.nagios.com/kb/article/n ... g-579.html which has some steps to run a raw query directly with /usr/bin/wmic. I'd also try increasing the debugging level using the --extrawmicarg and "--debuglevel=4" options as described in the same kb.

Did authentication protocols or requirements change on the Windows machine recently?

[N4g10s4dm1n]

I've already run raw queries and the result is the same.

I've also run Get-WmiObject directly on the Windows servers (even using the same account that is used in the Nagios auth file) and I get successful results.

Here is output from running a check with increased debugging.

Code: Select all

[root@myserver ~]# /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -u mydomain/\useraccount -p password --extrawmicarg "--debuglevel=4"
Command Line (v1.6): /usr/local/nagios/libexec/check_wmi_plus.pl -d -z -H myserver.mydomain -m checkcpu -u mydomain/useraccount -p password --extrawmicarg --debuglevel=4
Base Dir: /usr/local/nagios/libexec
Conf File Dir: /usr/local/nagios/libexec
Loaded Conf File /usr/local/nagios/libexec/check_wmi_plus.conf
Extra Wmic Arguments specified:--debuglevel=4
Starting Keep State Mode
STATE FILE: /tmp/cwpss_checkcpu__10106617___.state
Checking previous data's expiry - Timestamp 1549868800 vs Expiry After 1550686706 (Keep State Expiry setting is 3600sec)
Data has expired - getting data again
Round #1 of 1
QUERY: /usr/bin/wmic '--debuglevel=4' '-U' 'mydomain/useraccount%password' '--namespace' 'root/cimv2' '//myserver.mydomain' 'select PercentProcessorTime,Timestamp_Sys100NS from Win32_PerfRawData_PerfOS_Processor where Name="_Total"'
OUTPUT: [param/loadparm.c:587:init_globals()] Initialising global parameters
[param/loadparm.c:2462:lp_load()] lp_load: refreshing parameters from /dev/null
[param/params.c:556:pm_process()] params.c:pm_process() - Processing configuration file "/dev/null"
[param/loadparm.c:2471:lp_load()] pm_process() returned Yes
[param/loadparm.c:1343:lp_add_hidden()] adding hidden service IPC$
[param/loadparm.c:1343:lp_add_hidden()] adding hidden service ADMIN$
[auth/kerberos/krb5_init_context.c:388:smb_krb5_init_context()] krb5_init_context failed (Invalid argument)
[auth/auth.c:447:auth_register()] AUTH backend 'winbind_samba3' registered
[auth/auth.c:447:auth_register()] AUTH backend 'winbind' registered
[auth/auth.c:447:auth_register()] AUTH backend 'name_to_ntstatus' registered
[auth/auth.c:447:auth_register()] AUTH backend 'fixed_challenge' registered
[auth/auth.c:447:auth_register()] AUTH backend 'unix' registered
[auth/auth.c:447:auth_register()] AUTH backend 'anonymous' registered
[auth/auth.c:447:auth_register()] AUTH backend 'sam' registered
[auth/auth.c:447:auth_register()] AUTH backend 'sam_ignoredomain' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'krb5' registered
[auth/gensec/gensec.c:1205:gensec_register()] gensec subsystem fake_gssapi_krb5 is disabled
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'schannel' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'spnego' registered
[auth/gensec/gensec.c:1205:gensec_register()] gensec subsystem gssapi_spnego is disabled
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'gssapi_krb5' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'gssapi_krb5_sasl' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'ntlmssp' registered
[lib/com/dcom/main.c:528:dcom_determine_rpc_binding()] Using binding ncacn_ip_tcp:myserver.mydomain
[librpc/rpc/dcerpc_connect.c:513:continue_map_binding()] Mapped to DCERPC endpoint 135
[lib/com/dcom/main.c:413:determine_rpc_binding_continue2()] dcerpc_ndr_request_recv returned NT_STATUS_OK
[lib/com/dcom/main.c:417:determine_rpc_binding_continue2()] IObjectExporter::ServerAlive returned NT_STATUS_OK
[auth/gensec/gensec_gssapi.c:304:gensec_gssapi_client_start()] Cannot do GSSAPI to an IP address
[auth/gensec/gensec.c:606:gensec_start_mech()] Failed to start GENSEC client mech gssapi_krb5: NT_STATUS_INVALID_PARAMETER
[auth/ntlmssp/ntlmssp_client.c:128:ntlmssp_client_challenge()] Got challenge flags:
[auth/ntlmssp/ntlmssp.c:72:debug_ntlmssp_flags()] Got NTLMSSP neg_flags=0x62898205
  NTLMSSP_NEGOTIATE_UNICODE
  NTLMSSP_REQUEST_TARGET
  NTLMSSP_NEGOTIATE_NTLM
  NTLMSSP_NEGOTIATE_ALWAYS_SIGN
  NTLMSSP_NEGOTIATE_NTLM2
  NTLMSSP_CHAL_TARGET_INFO
  NTLMSSP_NEGOTIATE_128
  NTLMSSP_NEGOTIATE_KEY_EXCH
[auth/ntlmssp/ntlmssp_client.c:242:ntlmssp_client_challenge()] NTLMSSP: Set final flags:
[auth/ntlmssp/ntlmssp.c:72:debug_ntlmssp_flags()] Got NTLMSSP neg_flags=0x60088205
  NTLMSSP_NEGOTIATE_UNICODE
  NTLMSSP_REQUEST_TARGET
  NTLMSSP_NEGOTIATE_NTLM
  NTLMSSP_NEGOTIATE_ALWAYS_SIGN
  NTLMSSP_NEGOTIATE_NTLM2
  NTLMSSP_NEGOTIATE_128
  NTLMSSP_NEGOTIATE_KEY_EXCH
[librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

Could not find the CLASS: line - an error occurred
WMI DATA:$VAR1 = undef;
UNKNOWN - The WMI query had problems. You might have your username/password wrong or the user's access level is too low.  Your Authentication File might be incorrectly formatted or inaccessible. Wmic error text on the next line.
[param/loadparm.c:587:init_globals()] Initialising global parameters
[param/loadparm.c:2462:lp_load()] lp_load: refreshing parameters from /dev/null
[param/params.c:556:pm_process()] params.c:pm_process() - Processing configuration file "/dev/null"
[param/loadparm.c:2471:lp_load()] pm_process() returned Yes
[param/loadparm.c:1343:lp_add_hidden()] adding hidden service IPC$
[param/loadparm.c:1343:lp_add_hidden()] adding hidden service ADMIN$
[auth/kerberos/krb5_init_context.c:388:smb_krb5_init_context()] krb5_init_context failed (Invalid argument)
[auth/auth.c:447:auth_register()] AUTH backend 'winbind_samba3' registered
[auth/auth.c:447:auth_register()] AUTH backend 'winbind' registered
[auth/auth.c:447:auth_register()] AUTH backend 'name_to_ntstatus' registered
[auth/auth.c:447:auth_register()] AUTH backend 'fixed_challenge' registered
[auth/auth.c:447:auth_register()] AUTH backend 'unix' registered
[auth/auth.c:447:auth_register()] AUTH backend 'anonymous' registered
[auth/auth.c:447:auth_register()] AUTH backend 'sam' registered
[auth/auth.c:447:auth_register()] AUTH backend 'sam_ignoredomain' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'krb5' registered
[auth/gensec/gensec.c:1205:gensec_register()] gensec subsystem fake_gssapi_krb5 is disabled
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'schannel' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'spnego' registered
[auth/gensec/gensec.c:1205:gensec_register()] gensec subsystem gssapi_spnego is disabled
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'gssapi_krb5' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'gssapi_krb5_sasl' registered
[auth/gensec/gensec.c:1229:gensec_register()] GENSEC backend 'ntlmssp' registered
[lib/com/dcom/main.c:528:dcom_determine_rpc_binding()] Using binding ncacn_ip_tcp:myserver.mydomain
[librpc/rpc/dcerpc_connect.c:513:continue_map_binding()] Mapped to DCERPC endpoint 135
[lib/com/dcom/main.c:413:determine_rpc_binding_continue2()] dcerpc_ndr_request_recv returned NT_STATUS_OK
[lib/com/dcom/main.c:417:determine_rpc_binding_continue2()] IObjectExporter::ServerAlive returned NT_STATUS_OK
[auth/gensec/gensec_gssapi.c:304:gensec_gssapi_client_start()] Cannot do GSSAPI to an IP address
[auth/gensec/gensec.c:606:gensec_start_mech()] Failed to start GENSEC client mech gssapi_krb5: NT_STATUS_INVALID_PARAMETER
[auth/ntlmssp/ntlmssp_client.c:128:ntlmssp_client_challenge()] Got challenge flags:
[auth/ntlmssp/ntlmssp.c:72:debug_ntlmssp_flags()] Got NTLMSSP neg_flags=0x62898205
  NTLMSSP_NEGOTIATE_UNICODE
  NTLMSSP_REQUEST_TARGET
  NTLMSSP_NEGOTIATE_NTLM
  NTLMSSP_NEGOTIATE_ALWAYS_SIGN
  NTLMSSP_NEGOTIATE_NTLM2
  NTLMSSP_CHAL_TARGET_INFO
  NTLMSSP_NEGOTIATE_128
  NTLMSSP_NEGOTIATE_KEY_EXCH
[auth/ntlmssp/ntlmssp_client.c:242:ntlmssp_client_challenge()] NTLMSSP: Set final flags:
[auth/ntlmssp/ntlmssp.c:72:debug_ntlmssp_flags()] Got NTLMSSP neg_flags=0x60088205
  NTLMSSP_NEGOTIATE_UNICODE
  NTLMSSP_REQUEST_TARGET
  NTLMSSP_NEGOTIATE_NTLM
  NTLMSSP_NEGOTIATE_ALWAYS_SIGN
  NTLMSSP_NEGOTIATE_NTLM2
  NTLMSSP_NEGOTIATE_128
  NTLMSSP_NEGOTIATE_KEY_EXCH
[librpc/rpc/dcerpc_util.c:1290:dcerpc_pipe_auth_recv()] Failed to bind to uuid REDACTED - NT_STATUS_NET_WRITE_FAULT
[librpc/rpc/dcerpc_connect.c:790:dcerpc_pipe_connect_b_recv()] failed NT status (c0000022) in dcerpc_pipe_connect_b_recv
[wmi/wmic.c:196:main()] ERROR: Login to remote object.
NTSTATUS: NT_STATUS_ACCESS_DENIED - Access denied

No authentication requirements have changed on the Windows side that I'm aware of.

Also, and this is just very odd and very frustrating, as a test I used system backups to restore both the Nagios box and a Windows box back to a point in time where I know everything was working (about 10 days ago) and I'm still getting the WMI failures. I've verified, again, that no network issues exist and the queries are getting through the Windows firewall.

[N4g10s4dm1n]

I've done some more testing.

I created a local account on the Windows box and gave it admin privileges. I ran the same query as above using this new local admin account, but modified not to use the domain portion of course, and got the same results in the shell (no info and access denied). However, in the event viewer on the Windows side I can see successful logins.

Running the check from Nagios using the local account returns the same error in the GUI as with the domain account "The WMI query had problems. You might have your username/password wrong or the user's access level is too low. Your authentication file might be incorrectly formatted or inaccessible." However, there NO login events at all for the account used.

Officially ripping my hair out on this one!

[cdienger]

For the domain user, are there any restrictions they have on which computers they can sign in from? I've been able to reproduce the error message if the user cannot login from all computers. See screenshot for the option.