Port checking difficulties

[rdubya]

Moving right along;

Code: Select all

Sep 16 13:02:20 nexusfirewall nrpe[8411]: Connection from xxx.xxx.xxx.xxx port 44764
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Host address is in allowed_hosts
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Handling the connection...
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Host is asking for command 'check_nexus_listen' to be run...
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Running command: /usr/lib64/nagios/plugins/check_listen_tcp_udp.sh -p 8070 -P tcp
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Command completed with return code 3 and output:
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Return Code: 3, Output: NRPE: Unable to read output
Sep 16 13:02:20 nexusfirewall nrpe[8411]: Connection from P[▒T▒#177 closed.

With that error code I did some more googling and found a tip to add

Code: Select all

2>&1

to the end of the command definition in nrpe.cfg to give me insight into stderr

Code: Select all

[root@engmon ~]# /usr/local/nagios/libexec/check_nrpe -H xxx.xxx.xxx.xxx -c check_nexus_listen
sh: /usr/lib64/nagios/plugins/check_listen_tcp_udp.sh: /bin/sh: bad interpreter: Permission denied

It's interesting to see this permissions issue pop up only remotely. The plugins folder is set to 755 and I've changed ownership to nagios and then to nrpe with no effect. I even found a thread blaming windows character on the error ran the dos2unix command on the script, though that had no effect either.

[tgriep]

It looks like it is a shell issue that is causing the problem.
Try editing that script and change the top line from

Code: Select all

#!/bin/sh

to

Code: Select all

#!/bin/bash

Save it and test to see if that works.

[rdubya]

Yep, I had done that too but forgot to mention it. No change.

[tgriep]

There must be something on that system that is blocking the ability to open shells.

[rdubya]

I'd be inclined to agree if it weren't for the other plugins that worked.

[tgriep]

The other plugins may not be trying to open a shell so that is why they are working.
Also, try adding the following to the /etc/sudoers file is that are not already there and see if that helps.

Code: Select all

Defaults:nagios !requiretty
nagios ALL=NOPASSWD: /usr/lib64/nagios/plugins/check_listen_tcp_udp.sh

[rdubya]

Thanks tgriep, though this didn't help either. It's an absolute mystery.

I'm getting a lot of good responses here that make a lot of sense to me, but I think I'm ready to move on the other method. In a way it makes more sense since it's checking the port from outside of the server. It works for the most part but the web interface is giving some unexpected results. From the front page of this post, here's what I have done;
I was not able to resolve this and thought to change my method by checking the port from *outside* of the remote host with the pre-existing tcp_check command.
I am able to successfully check a port on the remote host from the nagios server;

Code: Select all

# /usr/local/nagios/libexec/check_tcp -H 10.177.177.220 -p 8070
TCP OK - 0.001 second response time on 10.177.177.220 port 8070|time=0.001022s;;;0.000000;10.000000

This is the pre-existing command definition in commands.cfg on the nagios server;

Code: Select all

# 'check_tcp' command definition
define command{
        command_name    check_tcp
   command_line    $USER1$/check_tcp -H $HOSTADDRESS$ -p $ARG1$ $ARG2$
        }

This is how the service is defined in localhost.cfg;

Code: Select all

define service{
        use                     local-service
        host_name               engmon
        service_description     tcp8070 on Nexusfirewall
        check_command           check_tcp!10.177.177.220!8070
}

This doesn't work, the services pages shows this failure;
tcp8070 on Nexusfirewall UNKNOWN 08-23-2016 15:06:04 0d 1h 3m 25s 4/4 check_tcp: Port must be a positive integer

I'm going to start playing with this again today, but any pointers would be gratefully appreciated.

[tgriep]

In your service check, you don't need to specify the IP address as the check will get it for you from the host_name object.
If you edit that service and change the check_command from

Code: Select all

check_command           check_tcp!10.177.177.220!8070

to

Code: Select all

check_command           check_tcp!8070!

that should fix it for you.

[rdubya]

Nevermind, found some info that pointed me to some healthy syntax and it set me up. I'm going to go crazy monitoring ports now.

(Yep, thanks tgriep)

[tgriep]

That is good to hear, and your welcome.
If you are good to go, shall I close and lockup this post as solved?