CHECK_NRPE: Received 0 bytes

[tgriep]

I know that the issue is intermittent, I am trying to see if I can get it to fail at all.

Did you increase the timeout for the check_nrpe command?
Add -t 115 to the command so it matches the timeout in the nsclient.ini file.

Nothing conclusive as to the cause of the issue in the nsclient.log file except the Virus_detected.vbs script is causing this error

Code: Select all

Failed to send data: The file handle supplied is not valid

Maybe that script is failing causing the others to not run.
Try disabling that check and see if that changes anything.

[gormank]

The filehandle error is the failure of the I/O script, not the virus. The log isn't sequential because the virus check never fails.

[tgriep]

Can you post how the check_nrpe command is defined in Nagios XI?

Can you post more of the nsclient.log file here so we can view it?
I would like to see from 2016-11-17 00:56:00 to 2016-11-17 01:05:00.
When the check runs normally, how long does it take?

[gormank]

I posted the entire nsclient.log. Not sure what more than that you need, or I can provide. ;) I guess you mean the nagios.log file.

Code: Select all

# grep -v ^# /usr/local/nagios/etc/services/CPU_Win_Queue.cfg

define service {
        service_description             CPU_Win_Queue
        use                             1VZW_default_service
        hostgroup_name                  1VZW_Windows_Physical,1VZW_Windows_Virtual
        check_command                   check_nrpe!check_cpu_queue!-a 4 6!!!!!!
        _xiwizard                       windowsserver
        register                        1
        }

# grep -v ^# /usr/local/nagios/etc/services/FS_Win_IO.cfg                                           
define service {
        service_description             FS_Win_IO
        use                             1VZW_default_service
        hostgroup_name                  1VZW_Windows_Physical,1VZW_Windows_Virtual
        display_name                    FS_Win_IO
        check_command                   check_nrpe!winfsio!!!!!!!
        register                        1
        }

[tgriep]

No, I don't need to see the nagios.log file.
I do need to see how the check_nrpe command is defined in the /usr/local/nagios/etc/commands.cfg file.
Man, that is the shortest nsclient.log file I have ever seen.
I have mine setup to truncate and it is 12 thousand lines long.

[gormank]

The nsclient.log file was truncated a few days ago since it had a few years worth of junk in it. It had thousands of lines of exactly the same text w/ different timestamps.

define command {
command_name check_nrpe
command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -u -t 120 -c $ARG1$ $ARG2$
}

[tgriep]

So far, everything looks good. I am thinking the script stop running at 115 seconds and then returns nothing for the check or that another script is crashing in the Windows system causing the same problem.
That is why I wanted to see more of the nsclient.log file.
One thing to try is to decrease the timeout in the check_nrpe command to 110 seconds and if the script is timing out in the Windows system, the XI server will timeout before it and set a critical alert.
If that happens, then we know it is an issue with the vbs scripts running on that system.

[gormank]

Ok,
I changed to 110 and will wait. The 120/115 is new. I had NRPE at 45 and the nsclient side at 30 (because I forgot t update them long ago) until a few days ago. Having the Windows side with a longer timeout is a new idea...
I also disabled the virus check and it doesn't seem to make a difference.

[tgriep]

How many times a day does it fail?
If it around the same time everyday?

[gormank]

You can see the intervals when it happens throughout the day. It happens all day long. The times look pretty random.

Code: Select all

# grep Win /usr/local/nagios/var/nagios.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008
[Thu Nov 17 00:59:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 01:50:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 02:40:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 03:31:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 04:21:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 05:12:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 06:03:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 07:27:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 07:44:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 09:08:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 10:50:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 12:14:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 15:20:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 16:11:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 17:01:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 17:52:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 19:17:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 20:58:33 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae

# grep Win /usr/local/nagios/var/nagios.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep FS_Win_IO
[Thu Nov 17 00:59:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 02:40:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 04:21:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 07:27:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 09:08:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 12:14:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 15:20:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 17:01:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 17:52:35 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 20:58:33 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae
[Thu Nov 17 22:22:43 2016] SERVICE ALERT: txslm2mwapp008;FS_Win_IO;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from dae

# grep Win /usr/local/nagios/var/nagios.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Thu Nov 17 01:50:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 03:31:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 05:12:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 06:03:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 07:44:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 10:50:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 16:11:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 17 19:17:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from

Here's a few days of the same alert. Note that 4 hosts do this on 2 services. Total number of alerts for this (not including ok) ranges from ~50-100 per day. Interestingly, in the last few daya since starting to try and figure it out by tweaking timeouts, etc., it seems worse. But that may be coincidence.

Code: Select all

# grep Win /usr/local/nagios/var/archives/nagios-11-11-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Thu Nov 10 05:23:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 10 10:24:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 10 16:15:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Thu Nov 10 20:26:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# grep Win /usr/local/nagios/var/archives/nagios-11-12-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Fri Nov 11 00:37:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Fri Nov 11 04:48:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Fri Nov 11 08:59:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Fri Nov 11 14:00:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Fri Nov 11 18:11:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Fri Nov 11 21:32:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# ^12^13
grep Win /usr/local/nagios/var/archives/nagios-11-13-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Sat Nov 12 00:53:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 03:24:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 05:55:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 08:26:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 11:47:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 14:18:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 16:49:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sat Nov 12 19:20:17 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# ^13^14
grep Win /usr/local/nagios/var/archives/nagios-11-14-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Sun Nov 13 00:56:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 06:32:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 08:13:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 09:54:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 11:35:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 13:16:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 14:57:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Sun Nov 13 19:43:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# ^14^15
grep Win /usr/local/nagios/var/archives/nagios-11-15-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Mon Nov 14 00:29:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 02:10:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 03:51:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 05:32:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 09:28:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 11:09:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 12:50:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 16:46:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 18:27:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Mon Nov 14 20:08:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# ^15^16
grep Win /usr/local/nagios/var/archives/nagios-11-16-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Tue Nov 15 02:19:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 07:05:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 08:46:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 10:27:08 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 12:08:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 15:14:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 16:55:18 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 22:16:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Tue Nov 15 23:57:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
# ^16^17
grep Win /usr/local/nagios/var/archives/nagios-11-17-2016-00.log | cut -c 1-105 | perl -pe 's/(\d+)/localtime($1)/e' | grep UNKNOWN | grep m2mwapp008 | grep CPU_Win_Queue
[Wed Nov 16 01:38:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 09:14:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 10:55:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 14:01:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 15:42:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 16:33:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from
[Wed Nov 16 21:54:05 2016] SERVICE ALERT: txslm2mwapp008;CPU_Win_Queue;UNKNOWN;SOFT;1;CHECK_NRPE: Received 0 bytes from