NRPE: Unable to read output

An open discussion forum for obtaining help with Nagios Core. Nagios Core users of all experience levels are welcome here. Subforum have been created for the discussion of Nagios Core and Nagios Plugin development.

NOTE: The SourceForge.net mailing lists have been deprecated in favor of this forum in order to expedite support and provide additional features not available on the old mailing list.

NRPE: Unable to read output

Postby Katya.Ailarov » Wed Mar 01, 2017 8:36 am

Hi Nagios support,

I'm facing issue with one specific plugin named: "check_jmx"

I have 3 machines:

1) Nagios server itself (ssbpops01_10.146.1.100)
2) Solaris remote host (Ssb-voor-01_168.162.97.143)
3) Solaris remote host (ssbpech01_10.146.1.38)

I installed this plugin on the 3 servers , only on Ssb-voor-01_168.162.97.143 it doesn't work even though the configuration, permissions are similar.
Code: Select all
root@ssb-voor-01 etc$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output

root@ssbpech01:/# /apps/nagios/libexec/check_nrpe -H 10.146.1.38 -c check_jmxmemory
JMX OK - TotalMemory = 3039821824 | 'TotalMemory'=3039821824;3247300864;3251495168;;


The permission of this plugin are the same on both servers:
Code: Select all
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ls -la
total 64
drwxrwxrwx   2 nagios   nagios         7 Mar  1 04:05 .
drwxrwxrwx   3 nagios   nagios        63 Mar  1 03:57 ..
-rwxrwxrwx   1 nagios   nagios     10351 Feb 15 04:40 LICENSE.txt
-rwxrwxrwx   1 nagios   nagios       749 Feb 15 04:40 README.txt
-rwxrwxrwx   1 nagios   nagios       282 Feb 15 04:40 check_jmx
-rwxrwxrwx   1 nagios   nagios     12925 Feb 15 04:40 check_jmx.jar
-rwxrwxrwx   1 nagios   nagios       206 Feb 28 09:56 jmx.cfg
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3


root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# ls -la
total 64
drwxrwxrwx   2 nagios   nagios         7 Jan 30 03:35 .
drwxrwxrwx   3 nagios   nagios        68 Jan 30 03:35 ..
-rwxrwxrwx   1 nagios   nagios     10351 Apr 19  2011 LICENSE.txt
-rwxrwxrwx   1 nagios   nagios       749 Apr 19  2011 README.txt
-rwxrwxrwx   1 nagios   nagios       282 Sep  5 12:07 check_jmx
-rwxrwxrwx   1 nagios   nagios     12925 Apr 19  2011 check_jmx.jar
-rwxrwxrwx   1 nagios   nagios       211 Apr 19  2011 jmx.cfg
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3#



if I'm running the command manually on voor-01_168.162.97.143 it is working perfectly:

Code: Select all
root@ssb-voor-01 etc$ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 3154640896 | 'TotalMemory'=3154640896;3247300864;3251495168;; 


But when the nrpe is running this it unable to read output

This is what I see in Nagios log:
Code: Select all
[1488365193] SERVICE NOTIFICATION: nagiosadmin;ssb-voor-01;JMX Memory;WARNING;notify-service-by-email;NRPE: Unable to read output


In Nagios server this is the relevant section for services.cfg:
Code: Select all
define service{
        use                     generic-service
        host_name               ssb-voor-01
        service_description     JMX Memory
        check_command           check_nrpe!check_jmxmemory
        check_interval          1
        }


Please see the attached nrpe.cfg file from voor-01 the server that the plugin not works.

Thanks in advanced,
Attachments
nrpe.cfg
nrpe.cfg
(8.4 KiB) Downloaded 10 times
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Postby mcapra » Wed Mar 01, 2017 4:46 pm

From ssb-voor-01_168.162.97.143, can you share the output of the following commands executed from the CLI:

Code: Select all
su nagios
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
exit

ps aux | grep nrpe
ps aux | grep xinetd


Are you able to run the check_nrpe command against ssb-voor-01_168.162.97.143 from the CLI of your Nagios Core machine? Like so:

Code: Select all
/path/to/plugins/check_nrpe -H 168.162.97.143 -c check_jmxmemory
Former Nagios employee - https://github.com/mcapra/
User avatar
mcapra
 
Posts: 2457
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Postby Katya.Ailarov » Thu Mar 02, 2017 6:31 am

[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2940731392 | 'TotalMemory'=2940731392;3247300864;3251495168;;

root@ssb-voor-01 ~$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 7946 6002 0 06:25:42 pts/7 0:00 grep nrpe

root@ssb-voor-01 ~$ ps -ef | grep xinetd
root 7966 6002 0 06:25:58 pts/7 0:00 grep xinetd
BTW, my nagios not runing with xinetd

root@ssb-voor-01 ~$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
root@ssb-voor-01 ~$
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Postby Katya.Ailarov » Thu Mar 02, 2017 10:29 am

[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2943877120 | 'TotalMemory'=2943877120;3247300864;3251495168;;
[nagios@ssb-voor-01]:~ $ exit
logout
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 1761 6002 0 10:25:23 pts/7 0:00 grep nrpe
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep xinetd
root 2038 6002 0 10:26:16 pts/7 0:00 grep xinetd
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
xinetd--> my nagios not running in xinetd

from my nagios server:
-bash-4.1# hostname
ssbpops01.ssb.sungard.com
-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1#

also i see something on the remote that is not working:
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
Sorry, user root is not allowed to execute '/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168' as nagios on ssb-voor-01.
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$


on the remote host that working:
root@ssbpech01:/# sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.38:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2999451648 | 'TotalMemory'=2999451648;3247300864;3251495168;;
root@ssbpech01:/#
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Postby mcapra » Thu Mar 02, 2017 3:40 pm

From the Nagios Core machine, can you try executing this again:

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory


Then, shortly after on 168.162.97.143, share the output of:

Code: Select all
tail -n 100 /var/log/messages | grep nrpe
Former Nagios employee - https://github.com/mcapra/
User avatar
mcapra
 
Posts: 2457
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Postby Katya.Ailarov » Sun Mar 05, 2017 7:48 am

Code: Select all
-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1# hostname
ssbpops01.ssb.sungard.com



Mar  4 03:10:32 ssb-voor-01 nrpe[26586]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:32 ssb-voor-01 nrpe[26587]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26594]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26595]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:33 ssb-voor-01 nrpe[26596]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:38 ssb-voor-01 nrpe[26607]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:10:38 ssb-voor-01 nrpe[26612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26859]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26860]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26864]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26865]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:32 ssb-voor-01 nrpe[26872]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:38 ssb-voor-01 nrpe[26880]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:11:38 ssb-voor-01 nrpe[26881]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27118]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27119]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:32 ssb-voor-01 nrpe[27123]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:33 ssb-voor-01 nrpe[27131]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:33 ssb-voor-01 nrpe[27132]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:38 ssb-voor-01 nrpe[27140]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:12:38 ssb-voor-01 nrpe[27145]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27343]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27344]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27345]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27346]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:32 ssb-voor-01 nrpe[27350]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:38 ssb-voor-01 nrpe[27364]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:13:38 ssb-voor-01 nrpe[27369]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27604]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27613]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27614]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:32 ssb-voor-01 nrpe[27615]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:38 ssb-voor-01 nrpe[27625]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:14:39 ssb-voor-01 nrpe[27626]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27833]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27834]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27835]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27849]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:32 ssb-voor-01 nrpe[27850]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  4 03:15:38 ssb-voor-01 nrpe[27854]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
"syslog" 13221 lines, 1364034 characters
Mar  5 07:45:31 ssb-voor-01 nrpe[21186]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:31 ssb-voor-01 nrpe[21194]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:51 ssb-voor-01 nrpe[21200]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:45:51 ssb-voor-01 nrpe[21201]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:18 ssb-voor-01 nrpe[21396]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:23 ssb-voor-01 nrpe[21397]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21444]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21454]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21461]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:31 ssb-voor-01 nrpe[21469]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:32 ssb-voor-01 nrpe[21477]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:51 ssb-voor-01 nrpe[21478]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:46:51 ssb-voor-01 nrpe[21479]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21697]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21698]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21717]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:31 ssb-voor-01 nrpe[21718]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:32 ssb-voor-01 nrpe[21725]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:51 ssb-voor-01 nrpe[21735]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar  5 07:47:51 ssb-voor-01 nrpe[21736]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Last edited by tmcdonald on Mon Mar 06, 2017 10:43 am, edited 1 time in total.
Reason: Please use [code][/code] tags around long output
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Postby mcapra » Mon Mar 06, 2017 2:27 pm

You'll need 2 terminal sessions open for this. In the first one, run this (you may need to yum/apt install tcpdump):

Code: Select all
tcpdump -w /tmp/42740_1.pcap -i any 'port 3515'


Then, in the second session, run the check_nrpe command a few times:

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory


After it's failed a few times, go back to the first session and cancel the tcpdump command. Share the resulting /tmp/42740_1.pcap file with us to investigate.
Former Nagios employee - https://github.com/mcapra/
User avatar
mcapra
 
Posts: 2457
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Postby Katya.Ailarov » Tue Mar 07, 2017 10:15 am

Hi Guys,

it will take time till the UNIX admin will install it.

meanwhile do we have any workaround?
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Re: NRPE: Unable to read output

Postby mcapra » Tue Mar 07, 2017 5:03 pm

Not immediately unfortunately. Something is behaving outside the norm and the tcpdump will help identify roughly where things are breaking down. Looking at the raw data transmitted when the check_nrpe command is initiated will help narrow down if it's client-side or server-side.
Former Nagios employee - https://github.com/mcapra/
User avatar
mcapra
 
Posts: 2457
Joined: Thu May 05, 2016 3:54 pm

Re: NRPE: Unable to read output

Postby Katya.Ailarov » Thu Mar 09, 2017 2:50 am

BTW, it is definitally the clien side (the nagios server itself) please see evidnce for that:

out put from ssbpops server -Nagios server itself:
==========================================
check-jmx of 168.162.97.143 (the server that show me NRPE:unable to read output)
==========================================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX CRITICAL - Error opening RMI connection: Connection refused to host: 168.162.97.143; nested exception is:
java.net.ConnectException: Connection timed out

check_jmx of 10.146.1.37 the server that the same check_jmx is working
==============================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.37:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2982150144 | 'TotalMemory'=2982150144;3247300864;3251495168;;

please be awrae that i run this commands fron Nagios server itself

please suggest.
Katya.Ailarov
 
Posts: 11
Joined: Wed Aug 17, 2016 5:29 am

Next

Return to Nagios Core

Who is online

Users browsing this forum: Google [Bot] and 9 guests