Strange "Unknown" errors with check_wsc

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
kelemvor
Posts: 12
Joined: Wed Jun 05, 2013 3:47 pm

Strange "Unknown" errors with check_wsc

Post by kelemvor »

So we have a strange issue that I thought maybe someone had seen before.

Starting a week or so ago, we're getting a lot of "Unknown" errors from Nagios. They all say something like:
check_wsc UNKNOWN: Problem getting service response message, code=500, message=read failed: Connection reset by peer
We apparently have Nagios setup so that our Windows Domain Controller handles the actual SNMP polling of the servers since it can see them all, and Nagios pulls its info from the DC.
We noticed that if someone remotes into the DC and logs on, the Unknown errors all stop. Then as soon as that person logs off so no one is actively remote into the machine, the Unknown errors come back. It's as if remoting into the machine wakes it up from a half asleep state so the communication starts working better.

We've had this up and running for years and this just started happening a week or so ago which happens to coincide with some Windows Patching we did. However, I don't see anything in the patch descriptions that it would have this affect on what we're doing.

Has anyone seen anything like this before and do you have any idea what might be causing it and maybe how to fix it? Normally it just gets the Unknown error once and the next check will work but sometimes it fails 2 or 3 times in a row which trigger an alert email to go out. We'd obviously not have people get woken up at night for no reason. ;)
slansing
Posts: 7698
Joined: Mon Apr 23, 2012 4:28 pm
Location: Travelling through time and space...

Re: Strange "Unknown" errors with check_wsc

Post by slansing »

Strange, do you by any chance have a text listing of the patches you pushed to these machines? What version if windows are these systems running on, are they all on the same version?
kelemvor
Posts: 12
Joined: Wed Jun 05, 2013 3:47 pm

Re: Strange "Unknown" errors with check_wsc

Post by kelemvor »

According to Powershell, the patches that got installed were:

Security Update KB2830290 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2829530 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2840149 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2772930 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2847204 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2829361 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2813170 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2804579 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Update KB2798162 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Update KB2820331 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2820197 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM
Security Update KB2813347 NT AUTHORITY\SYSTEM 5/26/2013 12:00:00 AM

Windows Update History also has:
KB890830
KB2804576

The DC is running Server 2008 R2 if that matters. :)

Thanks.
sreinhardt
-fno-stack-protector
Posts: 4366
Joined: Mon Nov 19, 2012 12:10 pm

Re: Strange "Unknown" errors with check_wsc

Post by sreinhardt »

Just a quick note, I am just beginning to take a look at this, however the check_wsc does not use snmp but instead wmi queries. If any of these updates effect wmi or .net framework this is likely our culprit. I will begin looking through them.
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
sreinhardt
-fno-stack-protector
Posts: 4366
Joined: Mon Nov 19, 2012 12:10 pm

Re: Strange "Unknown" errors with check_wsc

Post by sreinhardt »

I would suggest looking at removing theses patches, in this order and testing one by one. Most likely this is an issue with the dot net xml issue fix, and neither of the other two. However I also included the rdp one as you mentioned it specifically runs fine with a user logged in, and the AD one as you mentioned this specifically runs on a DC.

KB2804576
http://technet.microsoft.com/security/bulletin/MS13-040
dot net 2sp2-4.5 xml spoofing ***

Security Update KB2813347
http://technet.microsoft.com/security/bulletin/MS13-029
RDP RCE fix

Security Update KB2772930
http://technet.microsoft.com/security/bulletin/MS13-032
Active directory DOS issues ***

Additionally I wanted to ask if the user that gets logged in via rdp also happens to be the same one running .net app?
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
Nav
Posts: 3
Joined: Fri Jun 07, 2013 12:35 pm

Re: Strange "Unknown" errors with check_wsc

Post by Nav »

I'm seeing the same issue but it doesn't look like we applied any of the updates you listed. I'm running version 2.9.
sreinhardt
-fno-stack-protector
Posts: 4366
Joined: Mon Nov 19, 2012 12:10 pm

Re: Strange "Unknown" errors with check_wsc

Post by sreinhardt »

2.9 of what? Can you please create a separate thread, I do not want to hijack kelemvor's thread and not resolve that issue.
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
kelemvor
Posts: 12
Joined: Wed Jun 05, 2013 3:47 pm

Re: Strange "Unknown" errors with check_wsc

Post by kelemvor »

I've passed those patches to the guy on my team who's looking into this. As for the ID logging on, any of us can remote into the DC with our own IDs and the problem goes away. It's not related to any specific user.

Thanks
sreinhardt
-fno-stack-protector
Posts: 4366
Joined: Mon Nov 19, 2012 12:10 pm

Re: Strange "Unknown" errors with check_wsc

Post by sreinhardt »

Interesting, let us know what they find, if I get a chance I might do some testing over the weekend.
Nagios-Plugins maintainer exclusively, unless you have other C language bugs with open-source nagios projects, then I am happy to help! Please pm or use other communication to alert me to issues as I no longer track the forum.
Nav
Posts: 3
Joined: Fri Jun 07, 2013 12:35 pm

Re: Strange "Unknown" errors with check_wsc

Post by Nav »

Nagios 2.9 I believe. Either way I'll just hang back and lurk I think we appear to have the same issue.
Locked