check_ovm3 error

[lgaddam]

Team,

We are getting below error while using check_ovm3 plugin. Please help me how to rectify the issue.
All are critical alerts around 30 services were created for this.

[nagios@01 ~]$ /usr/local/nagios/libexec/check_ovm3 -H xx.xx.xx.xx -u admin -p xxxxxx -a filesystem -n fs_Preproduccion3 -w 92 -c 95
CRITICAL - Login not possible! Timeout 15 Try ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 10000 [email protected]

[jdunitz]

Having looked through the code of the check_ovm3 plugin, it appears to use expect to ssh to the OVM server and look for certain prompts.
If those are missing or different, it will fail. Also, if ssh is not accessible for some reason, it will fail.

So, the error is suggesting that you try ssh'ing manually. Did you try that? What happens when you do

Code: Select all

 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 10000 [email protected]

Does it work, or are you unable to log in that way?

[lgaddam]

HI,
Im not getting any output , waited for10 mins but its blank,
it didnt get out of the prompt or got any error.

[root@p01 ~]# ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 10000 [email protected]

[jdunitz]

It looks like maybe nothing is listening on port 10000, or possibly that entire machine is unreachable for some reason.

There are a few things to do to troubleshoot this further.

First, just try a regular SSH to that machine:

Code: Select all

ssh  -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null  [email protected]

You should be seeing dozens of lines of debug output about how SSH is connecting, and key exchanges and whatnot.

If you don't see that, it might not be listening to SSH connections, or may be blocked by a firewall, etc.

You can also try telnetting to port 10000 just to see if it connects

Code: Select all

# telnet xx.xx.xx.xx 10000

And you should see something like:

Code: Select all

# telnet 192.168.0.120 10000
Trying 192.168.0.120...
Connected to 192.168.0.120.
Escape character is '^]'.

Also, you can install nmap (if you don't have it already; you might also need to install SVN on some systems to get it to work):

Code: Select all

 
# yum install svn ; yum install nmap 

...then you can use that to scan port 22 and/or 10000 and see if it's listening:

Code: Select all

[root@localhost tmp]# nmap -p 22 192.168.0.5

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-02 09:22 CST
Nmap scan report for 192.168.0.5
Host is up (0.00028s latency).
PORT   STATE SERVICE
22/tcp open  ssh       <===== THIS ONE IS LISTENING because it says "open"
MAC Address: 00:0C:29:01:6C:62 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 0.05 seconds
[root@localhost tmp]# nmap -v -p 22 192.168.0.5

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-02 09:22 CST
Initiating ARP Ping Scan at 09:22
Scanning 192.168.0.5 [1 port]
Completed ARP Ping Scan at 09:22, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 09:22
Completed Parallel DNS resolution of 1 host. at 09:22, 0.00s elapsed
Initiating SYN Stealth Scan at 09:22
Scanning 192.168.0.5 [1 port]
Discovered open port 22/tcp on 192.168.0.5
Completed SYN Stealth Scan at 09:22, 0.00s elapsed (1 total ports)
Nmap scan report for 192.168.0.5
Host is up (0.00018s latency).
PORT   STATE SERVICE
22/tcp open  ssh
MAC Address: 00:0C:29:01:6C:62 (VMware)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds
           Raw packets sent: 2 (72B) | Rcvd: 2 (72B)
[root@localhost tmp]# nmap -v -p 2222 192.168.0.5

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-02 09:23 CST
Initiating ARP Ping Scan at 09:23
Scanning 192.168.0.5 [1 port]
Completed ARP Ping Scan at 09:23, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 09:23
Completed Parallel DNS resolution of 1 host. at 09:23, 0.00s elapsed
Initiating SYN Stealth Scan at 09:23
Scanning 192.168.0.5 [1 port]
Completed SYN Stealth Scan at 09:23, 0.00s elapsed (1 total ports)
Nmap scan report for 192.168.0.5
Host is up (0.00025s latency).
PORT     STATE  SERVICE
2222/tcp closed EtherNet/IP-1  <===== THIS ONE IS NOT LISTENING because it says "closed"
MAC Address: 00:0C:29:01:6C:62 (VMware)

Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds
           Raw packets sent: 2 (72B) | Rcvd: 2 (68B)
[root@localhost tmp]#

Hope that helps!

[lgaddam]

Below is the outputs.

Below output got stuck at last line and I have exited forcefully from that.

[root@p01 ~]# ssh -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 10000 [email protected]
OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug1: Connecting to 192.168.5.200 [192.168.5.200] port 10000.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/identity type -1
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_dsa type 2
debug1: loaded 3 keys


[root@glnagiosp01 ~]# telnet 192.168.5.200 10000
Trying 192.168.5.200...
Connected to 192.168.5.200 (192.168.5.200).
Escape character is '^]'.
^]
telnet> close
Connection closed.


[root@glnagiosp01 ~]# nmap -p 22 192.168.5.200

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-03 11:16 CET
Nmap scan report for 192.168.5.200
Host is up (0.00033s latency).
PORT STATE SERVICE
22/tcp open ssh

Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds



root@glnagiosp01 ~]# nmap -p 10000 192.168.5.200

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-03 11:17 CET
Nmap scan report for 192.168.5.200
Host is up (0.00042s latency).
PORT STATE SERVICE
10000/tcp filtered snet-sensor-mgmt

Nmap done: 1 IP address (1 host up) scanned in 0.39 seconds

[jdunitz]

Are you able to see anything in the logs on the remote (192.168.5.200) side when you try to connect via ssh to either port 22 or 10000?

Good places to look are /var/log/messages and /var/log/secure, and /var/log/auth.log (if your system has it).

We want to see how far you're getting with the connection. The results of the nmap check are showing that port 10000 is filtered, so make sure there is nothing blocking (firewall) connection between Nagios XI and the host.

At this point, we're not really debugging anything specific to Nagios, but the behavior and interaction between the ovm3 plugin, SSH, and whatever unique setup you have in your network (firewalls, filtering, etc.).

[lgaddam]

Im not able to see any error in /var/log/messages.

[root@glnagiosp01 ~]# nmap -p 10000 192.168.5.200

Starting Nmap 6.47 ( http://nmap.org ) at 2020-03-11 08:47 CET
Nmap scan report for 192.168.5.200
Host is up (0.00052s latency).
PORT STATE SERVICE
10000/tcp open snet-sensor-mgmt

Nmap done: 1 IP address (1 host up) scanned in 0.16 seconds
[root@glnagiosp01 ~]#


Port 10000 is not showing filtered as earlier , now it showing as open in the above output.
Linux team saying everything looks good from there end. Earlier it was working, now it was not working.

We have to figure out what is causing these issues and asking us to enable monitoring.

[jdunitz]

Can you try SSHing to port 10000 from the OVM machine itself?

Code: Select all

# ssh admin@localhost -p 10000

Does that work?

If so, can you try SSHing to port 10000 from another machine (a PC, another linux server, whatever...) that is on the same network segment as your OVM machine?

We're just trying to troubleshoot the basic connectivity to the OVM CLI at this point.