NRPE: Unable to read output

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

NRPE: Unable to read output

Post by Katya.Ailarov »

Hi Nagios support,

I'm facing issue with one specific plugin named: "check_jmx"

I have 3 machines:

1) Nagios server itself (ssbpops01_10.146.1.100)
2) Solaris remote host (Ssb-voor-01_168.162.97.143)
3) Solaris remote host (ssbpech01_10.146.1.38)

I installed this plugin on the 3 servers , only on Ssb-voor-01_168.162.97.143 it doesn't work even though the configuration, permissions are similar.

Code: Select all

root@ssb-voor-01 etc$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output 

root@ssbpech01:/# /apps/nagios/libexec/check_nrpe -H 10.146.1.38 -c check_jmxmemory
JMX OK - TotalMemory = 3039821824 | 'TotalMemory'=3039821824;3247300864;3251495168;; 
The permission of this plugin are the same on both servers:

Code: Select all

 root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ls -la
total 64
drwxrwxrwx   2 nagios   nagios         7 Mar  1 04:05 .
drwxrwxrwx   3 nagios   nagios        63 Mar  1 03:57 ..
-rwxrwxrwx   1 nagios   nagios     10351 Feb 15 04:40 LICENSE.txt
-rwxrwxrwx   1 nagios   nagios       749 Feb 15 04:40 README.txt
-rwxrwxrwx   1 nagios   nagios       282 Feb 15 04:40 check_jmx
-rwxrwxrwx   1 nagios   nagios     12925 Feb 15 04:40 check_jmx.jar
-rwxrwxrwx   1 nagios   nagios       206 Feb 28 09:56 jmx.cfg
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3 


root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# ls -la
total 64
drwxrwxrwx   2 nagios   nagios         7 Jan 30 03:35 .
drwxrwxrwx   3 nagios   nagios        68 Jan 30 03:35 ..
-rwxrwxrwx   1 nagios   nagios     10351 Apr 19  2011 LICENSE.txt
-rwxrwxrwx   1 nagios   nagios       749 Apr 19  2011 README.txt
-rwxrwxrwx   1 nagios   nagios       282 Sep  5 12:07 check_jmx
-rwxrwxrwx   1 nagios   nagios     12925 Apr 19  2011 check_jmx.jar
-rwxrwxrwx   1 nagios   nagios       211 Apr 19  2011 jmx.cfg
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# 

if I'm running the command manually on voor-01_168.162.97.143 it is working perfectly:

Code: Select all

root@ssb-voor-01 etc$ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 3154640896 | 'TotalMemory'=3154640896;3247300864;3251495168;;  
But when the nrpe is running this it unable to read output

This is what I see in Nagios log:

Code: Select all

[1488365193] SERVICE NOTIFICATION: nagiosadmin;ssb-voor-01;JMX Memory;WARNING;notify-service-by-email;NRPE: Unable to read output 
In Nagios server this is the relevant section for services.cfg:

Code: Select all

define service{
        use                     generic-service
        host_name               ssb-voor-01
        service_description     JMX Memory
        check_command           check_nrpe!check_jmxmemory
        check_interval          1
        } 
Please see the attached nrpe.cfg file from voor-01 the server that the plugin not works.

Thanks in advanced,
Attachments
nrpe.cfg
nrpe.cfg
(8.4 KiB) Downloaded 283 times
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Post by mcapra »

From ssb-voor-01_168.162.97.143, can you share the output of the following commands executed from the CLI:

Code: Select all

su nagios
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
exit

ps aux | grep nrpe
ps aux | grep xinetd
Are you able to run the check_nrpe command against ssb-voor-01_168.162.97.143 from the CLI of your Nagios Core machine? Like so:

Code: Select all

/path/to/plugins/check_nrpe -H 168.162.97.143 -c check_jmxmemory
Former Nagios employee
https://www.mcapra.com/
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Post by Katya.Ailarov »

[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2940731392 | 'TotalMemory'=2940731392;3247300864;3251495168;;

root@ssb-voor-01 ~$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 7946 6002 0 06:25:42 pts/7 0:00 grep nrpe

root@ssb-voor-01 ~$ ps -ef | grep xinetd
root 7966 6002 0 06:25:58 pts/7 0:00 grep xinetd
BTW, my nagios not runing with xinetd

root@ssb-voor-01 ~$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
root@ssb-voor-01 ~$
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Post by Katya.Ailarov »

[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2943877120 | 'TotalMemory'=2943877120;3247300864;3251495168;;
[nagios@ssb-voor-01]:~ $ exit
logout
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 1761 6002 0 10:25:23 pts/7 0:00 grep nrpe
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep xinetd
root 2038 6002 0 10:26:16 pts/7 0:00 grep xinetd
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
xinetd--> my nagios not running in xinetd

from my nagios server:
-bash-4.1# hostname
ssbpops01.ssb.sungard.com
-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1#

also i see something on the remote that is not working:
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
Sorry, user root is not allowed to execute '/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168' as nagios on ssb-voor-01.
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$


on the remote host that working:
root@ssbpech01:/# sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.38:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2999451648 | 'TotalMemory'=2999451648;3247300864;3251495168;;
root@ssbpech01:/#
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Post by mcapra »

From the Nagios Core machine, can you try executing this again:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
Then, shortly after on 168.162.97.143, share the output of:

Code: Select all

tail -n 100 /var/log/messages | grep nrpe
Former Nagios employee
https://www.mcapra.com/
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Post by Katya.Ailarov »

Code: Select all

-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1# hostname
ssbpops01.ssb.sungard.com



Mar  4 03:10:32 ssb-voor-01 nrpe[26586]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:32 ssb-voor-01 nrpe[26587]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26594]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26595]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26596]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:38 ssb-voor-01 nrpe[26607]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:38 ssb-voor-01 nrpe[26612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26859]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26860]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26864]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26865]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26872]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:38 ssb-voor-01 nrpe[26880]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:38 ssb-voor-01 nrpe[26881]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27118]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27119]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27123]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:33 ssb-voor-01 nrpe[27131]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:33 ssb-voor-01 nrpe[27132]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:38 ssb-voor-01 nrpe[27140]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:38 ssb-voor-01 nrpe[27145]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27343]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27344]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27345]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27346]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27350]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:38 ssb-voor-01 nrpe[27364]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:38 ssb-voor-01 nrpe[27369]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27604]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27613]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27614]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27615]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:38 ssb-voor-01 nrpe[27625]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:39 ssb-voor-01 nrpe[27626]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27833]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27834]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27835]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27849]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27850]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:38 ssb-voor-01 nrpe[27854]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
"syslog" 13221 lines, 1364034 characters
Mar  5 07:45:31 ssb-voor-01 nrpe[21186]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:31 ssb-voor-01 nrpe[21194]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:51 ssb-voor-01 nrpe[21200]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:51 ssb-voor-01 nrpe[21201]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:18 ssb-voor-01 nrpe[21396]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:23 ssb-voor-01 nrpe[21397]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21444]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21454]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21461]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21469]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:32 ssb-voor-01 nrpe[21477]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:51 ssb-voor-01 nrpe[21478]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:51 ssb-voor-01 nrpe[21479]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21697]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21698]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21717]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21718]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:32 ssb-voor-01 nrpe[21725]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:51 ssb-voor-01 nrpe[21735]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:51 ssb-voor-01 nrpe[21736]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Last edited by tmcdonald on Mon Mar 06, 2017 10:43 am, edited 1 time in total.
Reason: Please use [code][/code] tags around long output
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Post by mcapra »

You'll need 2 terminal sessions open for this. In the first one, run this (you may need to yum/apt install tcpdump):

Code: Select all

tcpdump -w /tmp/42740_1.pcap -i any 'port 3515'
Then, in the second session, run the check_nrpe command a few times:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
After it's failed a few times, go back to the first session and cancel the tcpdump command. Share the resulting /tmp/42740_1.pcap file with us to investigate.
Former Nagios employee
https://www.mcapra.com/
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Post by Katya.Ailarov »

Hi Guys,

it will take time till the UNIX admin will install it.

meanwhile do we have any workaround?
User avatar
mcapra
Posts: 3739
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Post by mcapra »

Not immediately unfortunately. Something is behaving outside the norm and the tcpdump will help identify roughly where things are breaking down. Looking at the raw data transmitted when the check_nrpe command is initiated will help narrow down if it's client-side or server-side.
Former Nagios employee
https://www.mcapra.com/
Katya.Ailarov
Posts: 25
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Post by Katya.Ailarov »

BTW, it is definitally the clien side (the nagios server itself) please see evidnce for that:

out put from ssbpops server -Nagios server itself:
==========================================
check-jmx of 168.162.97.143 (the server that show me NRPE:unable to read output)
==========================================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX CRITICAL - Error opening RMI connection: Connection refused to host: 168.162.97.143; nested exception is:
java.net.ConnectException: Connection timed out

check_jmx of 10.146.1.37 the server that the same check_jmx is working
==============================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.37:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2982150144 | 'TotalMemory'=2982150144;3247300864;3251495168;;

please be awrae that i run this commands fron Nagios server itself

please suggest.
Locked