check_xi_service_tcp socket timeout issue

[ericssonvietnam]

Hi

I have added a new check for the above check and increase the time out


COMMAND: /usr/local/nagios/libexec/check_tcp -H hnrbt10 -p 1521 -t 120
OUTPUT: connect to address hnrbt10 and port 1521: Connection timed out

but still getting timeout

[tgriep]

It could be that the server is not listening on that port or that the Nagios server is blocked from accessing it.
If you run the following command on the Nagios server as root, it should tell you if the port it open or closed.

Code: Select all

nmap hnrbt10 -p 1521

If it reports that the State of the port is closed, that could be why the check_tcp plugin is timing out.

[ericssonvietnam]

It could be that the server is not listening on that port or that the Nagios server is blocked from accessing it.
If you run the following command on the Nagios server as root, it should tell you if the port it open or closed.

Code: Select all

nmap hnrbt10 -p 1521

If it reports that the State of the port is closed, that could be why the check_tcp plugin is timing out.

Starting Nmap 5.51 ( http://nmap.org ) at 2017-05-31 00:53 ICT
Failed to find device eth2 which was referenced in /proc/net/route
Nmap scan report for hnrbt10 (10.10.165.17)
Host is up (0.00021s latency).
rDNS record for 10.10.165.17: HNRBT10
PORT STATE SERVICE
1521/tcp filtered oracle

Nmap done: 1 IP address (1 host up) scanned in 0.48 seconds

[avandemore]

It appears that port is not accepting connections from the XI system. Is the listener active? Is there a firewall involved?

[ericssonvietnam]

It appears that port is not accepting connections from the XI system. Is the listener active? Is there a firewall involved?

Can you suggest if the below two nodes have the snmp traffic restricted as we haven't received any trap from the server in past.

[root@nagiosxi ~]# nmap -Pn -sU -p U:161 10.10.165.38

Starting Nmap 5.51 ( http://nmap.org ) at 2017-06-02 18:08 ICT
Nmap scan report for HNSMS1-ST1 (10.10.165.38)
Host is up.
PORT STATE SERVICE
161/udp open|filtered snmp

Nmap done: 1 IP address (1 host up) scanned in 2.22 seconds
[root@nagiosxi ~]# nmap -Pn -sU -p U:161 10.10.165.39

Starting Nmap 5.51 ( http://nmap.org ) at 2017-06-02 18:08 ICT
Nmap scan report for HNSMS1-ST2 (10.10.165.39)
Host is up.
PORT STATE SERVICE
161/udp open|filtered snmp

Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds

What does the state open|filtered means ?
can you forward me any doc which can clarify me the nmap command options.

[avandemore]

You can find the meaning here:
https://nmap.org/book/man-port-scanning-techniques.html

I don't understand why you are running a udp check when your question is about a tcp connection. You need to run a tcp check if you want to find out the status of a tcp connection.

[ericssonvietnam]

You can find the meaning here:
https://nmap.org/book/man-port-scanning-techniques.html

I don't understand why you are running a udp check when your question is about a tcp connection. You need to run a tcp check if you want to find out the status of a tcp connection.

Just wanted to understand what does the output of nmap says about the node.

As i am not getting any traps from the node ::

[root@nagiosxi ~]# nmap -Pn -sU -p U:161 10.10.165.39

Starting Nmap 5.51 ( http://nmap.org ) at 2017-06-02 18:08 ICT
Nmap scan report for HNSMS1-ST2 (10.10.165.39)
Host is up.
PORT STATE SERVICE
161/udp open|filtered snmp

Nmap done: 1 IP address (1 host up) scanned in 2.07 seconds

What does the state open|filtered means ? in the above output.

[avandemore]

From https://nmap.org/book/man-port-scanning-techniques.html:

If no response is received after retransmissions, the port is classified as open|filtered. This means that the port could be open, or perhaps packet filters are blocking the communication.

Another way of saying it is that command provides almost no insight into the issue.

What about your tcp issue? Is that resolved?

[ericssonvietnam]

From https://nmap.org/book/man-port-scanning-techniques.html:

If no response is received after retransmissions, the port is classified as open|filtered. This means that the port could be open, or perhaps packet filters are blocking the communication.

Another way of saying it is that command provides almost no insight into the issue.

What about your tcp issue? Is that resolved?

In eventlogs i am recieving the below message for the SNMP trap
Node name = 'HNSMS1-ST2' IP Address 10.10.165.39

Runtime Warning 2017-06-03 02:08:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 2m 0s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-03 00:06:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 59s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service
Runtime Warning 2017-06-02 22:04:29 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 2m 1s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 20:02:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 59s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 18:00:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 2m 0s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 15:58:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 59s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 13:56:29 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 59s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 11:54:30 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 2m 1s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 09:52:29 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 0m 1s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 07:52:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 59s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 05:50:29 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 2m 0s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Runtime Warning 2017-06-02 03:48:28 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 0m 24s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Verification Warning 2017-06-02 02:04:27 Warning: Service 'SNMP Traps' on host 'HNSMS1-ST2' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.
Runtime Warning 2017-06-02 01:48:04 Warning: The results of service 'SNMP Traps' on host 'HNSMS1-ST2' are stale by 0d 0h 1m 10s (threshold=0d 2h 0m 0s). I'm forcing an immediate check of the service.
Verification Warning 2017-06-02 01:46:05 Warning: Service 'SNMP Traps' on host 'HNSMS1-ST2' has a notification interval less than its check interval! Notifications are only re-sent after checks are made, so the effective notification interval will be that of the check interval.

Traffic filtering for the Node name = 'HNSMS1-ST2' IP Address 10.10.165.39 is the reason for not receiving the trap at Nagios end.

FOR TCP ISSUE AM TAKING FOLLOWUP WITH THE NETWORK TEAM TO GET THE PORT OPEN AS THE PORT IS OPEN AND LISTENING ON THE CLIENT END.

[avandemore]

I'm having a difficult time understanding this thread. It was opened with a tcp timeout issue and has now crossed over into an SNMP trap issue. It isn't clear to me they are related in anyway. Can you please explain what your root issue issue is? If you have 2 or more seperate issues, please open a new ticket or thread for each. We like to keep one specific issue per thread or ticket for many reasons. However briefly you should at least include the definition of your SNMP trap object Service 'SNMP Traps' on host 'HNSMS1-ST2' if that indeed is an unrelated issue.