How do I not alert on Socket timeout after 30 seconds.

[tgriep]

I am not seeing an email Notification for a socket timeout.
The Notification is for the CPU Usage being over 95%.

There is an Alert for the timeout but no Notification for it.
Alerts and Notifications are 2 different things. See this.
https://www.youtube.com/watch?v=EDmZ6NtCH7s

About the Notification after 1.
Did you upgrade the version of XI on the server?
There are some bugs in core that could cause this issue.
FYI, there was a maintenance release for core yesterday that could fix the issue.
Do you want to try that?

Another cause could be a duplicate nagios process.
Run the following as root to ensure that is not happening.

Code: Select all

service nagios stop
killall -9 nagios
service nagios start

[gormank]

I'm not sure if it helps but I run into the occasionally...
I set the check_nrpe commands to tell it to set the alert to unknown rather than critical and not notify on unknown.
check_nrpe -t n:m ...
n is number of seconds to timeout
m is severity (1-3) and 3 is unknown

I think Nagios also has a global timeout for checks so if that's less than check_nrpe, it will be used.

[lmiltchev]

With NRPE v3, you could use the "-u" flag to return UNKNOWN instead of CRITICAL on connection issues.

-u, --unknown-timeout Make connection problems return UNKNOWN instead of CRITICAL

[mkeey]

My apologies for the excessive delay. A lot going on at work.
I like the suggestion of returning "unknown" with the "u" option. Not sure where this should be coded however. Can you provide some instructions?

[lmiltchev]

You can add the "-u" flag to the check_nrpe command.

CCM > Commands > check_nrpe > Edit > Save > Apply Configuration

Example:

Code: Select all

define command {
    command_name    check_nrpe
    command_line    $USER1$/check_nrpe -u -H $HOSTADDRESS$ -t 60 -c $ARG1$ $ARG2$
}

[mkeey]

Excellent! Thank you. Discussed with the team and manager. We'll be giving this a go on our Test and UAT systems. Please leave case open until I try this for a while.

[lmiltchev]

Sure. We will keep the topic open for now.

[mkeey]

Added the "-u" to the check_nrpe script via the XI CCM screen. Almost immediately things started going to an "unknown" status. But, only for our Linux servers. Windows still issued the Critical for socket timeouts and connection refused message.


SERVER01 (Windows)
CPU Usage
Critical 24d 23h 26m 23s 20/20 08/13/2019 10:44:19 CRITICAL - Socket timeout
Drive C: Disk Usage
Critical 24d 23h 26m 28s 05/05 08/13/2019 10:45:56 CRITICAL - Socket timeout
Memory Usage
Critical 24d 23h 25m 39s 24/24 08/13/2019 10:46:16 CRITICAL - Socket timeout


SERVER03 (Linux Cloud)
/ Disk Usage
Unknown 04d 22h 05m 57s 05/05 08/13/2019 10:45:21 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
/home Disk Usage
Unknown 04d 22h 06m 19s 05/05 08/13/2019 10:44:53 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
/opt Disk Usage
Unknown 04d 22h 10m 07s 05/05 08/13/2019 10:46:19 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
/tmp Disk Usage
Unknown 04d 22h 09m 48s 05/05 08/13/2019 10:46:28 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
/var Disk Usage
Unknown 04d 22h 07m 47s 05/05 08/13/2019 10:48:41 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
/var/log Disk Usage
Unknown 04d 22h 07m 14s 05/05 08/13/2019 10:49:13 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
CPU Stats
Unknown 04d 22h 06m 43s 20/20 08/13/2019 10:44:49 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused
Memory Usage
Unknown 04d 22h 02m 39s 24/24 08/13/2019 10:40:15 (No output on stdout) stderr: connect to address 10.225.73.14 port 5666: Connection refused


SERVER04 (Windows)
CPU Usage
Critical 11d 00h 56m 44s 20/20 08/13/2019 10:45:26 connect to address 10.13.7.8 and port 12489: Connection refused
Drive C: Disk Usage
Critical 11d 00h 59m 11s 05/05 08/13/2019 10:47:07 connect to address 10.13.7.8 and port 12489: Connection refused
Memory Usage
Critical 11d 00h 55m 33s 24/24 08/13/2019 10:44:25 connect to address 10.13.7.8 and port 12489: Connection refused


SERVER05 (Linux OnPrem)
/ Disk Usage
Unknown 04d 22h 8m 29s 05/05 08/13/2019 10:48:18 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/home Disk Usage
Unknown 04d 22h 10m 25s 05/05 08/13/2019 10:46:25 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/opt Disk Usage
Unknown 04d 22h 9m 30s 05/05 08/13/2019 10:47:17 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/srv/bit9 Disk Usage
Unknown 04d 22h 8m 29s 05/05 08/13/2019 10:48:21 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/tmp Disk Usage
Unknown 04d 22h 7m 53s 05/05 08/13/2019 10:48:50 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/var Disk Usage
Unknown 04d 22h 6m 37s 05/05 08/13/2019 10:45:23 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
/var/log Disk Usage
Unknown 04d 22h 5m 47s 05/05 08/13/2019 10:46:10 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
CPU Stats
Unknown 04d 22h 8m 25s 20/20 08/13/2019 10:48:30 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused
Memory Usage
Unknown 04d 22h 2m 17s 24/24 08/13/2019 10:40:49 (No output on stdout) stderr: connect to address 10.186.7.156 port 5666: Connection refused


SERVER06 (Linux OnPrem)
/ Disk Usage
Unknown 04d 22h 10m 29s 05/05 08/13/2019 10:46:15 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/home Disk Usage
Unknown 04d 22h 08m 54s 05/05 08/13/2019 10:48:03 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/opt Disk Usage
Unknown 04d 22h 08m 15s 05/05 08/13/2019 10:48:37 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/srv/bit9 Disk Usage
Unknown 04d 22h 09m 24s 05/05 08/13/2019 10:47:28 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/tmp Disk Usage
Unknown 04d 22h 08m 35s 05/05 08/13/2019 10:48:13 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/var Disk Usage
Unknown 04d 22h 07m 34s 05/05 08/13/2019 10:49:15 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
/var/log Disk Usage
Unknown 04d 22h 06m 54s 05/05 08/13/2019 10:44:58 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
CPU Stats
Unknown 04d 22h 05m 41s 20/20 08/13/2019 10:46:23 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused
Memory Usage
Unknown 04d 22h 04m 3s 24/24 08/13/2019 10:48:57 (No output on stdout) stderr: connect to address 10.186.7.158 port 5666: Connection refused

[lmiltchev]

Windows still issued the Critical for socket timeouts and connection refused message.

This is because you are not using check_nrpe for these checks but check_nt... check_nt has the same option, however it's deprecated, and it doesn't work.

Usage:
check_nt -H host -v variable [-p port] [-w warning] [-c critical]
[-l params] [-d SHOWALL] [-u](DEPRECATED) [-t timeout]

You may need to switch to using check_nrpe instead of check_nt. See the NSClient++ documentation here:

https://docs.nsclient.org/howto/nrpe/