Page 1 of 3
NRPE: Unable to read output
Posted: Wed Mar 01, 2017 8:36 am
by Katya.Ailarov
Hi Nagios support,
I'm facing issue with one specific plugin named: "check_jmx"
I have 3 machines:
1) Nagios server itself (ssbpops01_10.146.1.100)
2) Solaris remote host (Ssb-voor-01_168.162.97.143)
3) Solaris remote host (ssbpech01_10.146.1.38)
I installed this plugin on the 3 servers , only on Ssb-voor-01_168.162.97.143 it doesn't work even though the configuration, permissions are similar.
Code: Select all
root@ssb-voor-01 etc$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
root@ssbpech01:/# /apps/nagios/libexec/check_nrpe -H 10.146.1.38 -c check_jmxmemory
JMX OK - TotalMemory = 3039821824 | 'TotalMemory'=3039821824;3247300864;3251495168;;
The permission of this plugin are the same on both servers:
Code: Select all
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ls -la
total 64
drwxrwxrwx 2 nagios nagios 7 Mar 1 04:05 .
drwxrwxrwx 3 nagios nagios 63 Mar 1 03:57 ..
-rwxrwxrwx 1 nagios nagios 10351 Feb 15 04:40 LICENSE.txt
-rwxrwxrwx 1 nagios nagios 749 Feb 15 04:40 README.txt
-rwxrwxrwx 1 nagios nagios 282 Feb 15 04:40 check_jmx
-rwxrwxrwx 1 nagios nagios 12925 Feb 15 04:40 check_jmx.jar
-rwxrwxrwx 1 nagios nagios 206 Feb 28 09:56 jmx.cfg
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# ls -la
total 64
drwxrwxrwx 2 nagios nagios 7 Jan 30 03:35 .
drwxrwxrwx 3 nagios nagios 68 Jan 30 03:35 ..
-rwxrwxrwx 1 nagios nagios 10351 Apr 19 2011 LICENSE.txt
-rwxrwxrwx 1 nagios nagios 749 Apr 19 2011 README.txt
-rwxrwxrwx 1 nagios nagios 282 Sep 5 12:07 check_jmx
-rwxrwxrwx 1 nagios nagios 12925 Apr 19 2011 check_jmx.jar
-rwxrwxrwx 1 nagios nagios 211 Apr 19 2011 jmx.cfg
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3# pwd
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3
root@ssbpech01:/apps/nagios/libexec/nagios-jmx-plugin-1.2.3#
if I'm running the command manually on voor-01_168.162.97.143 it is working perfectly:
Code: Select all
root@ssb-voor-01 etc$ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 3154640896 | 'TotalMemory'=3154640896;3247300864;3251495168;;
But when the nrpe is running this it unable to read output
This is what I see in Nagios log:
Code: Select all
[1488365193] SERVICE NOTIFICATION: nagiosadmin;ssb-voor-01;JMX Memory;WARNING;notify-service-by-email;NRPE: Unable to read output
In Nagios server this is the relevant section for services.cfg:
Code: Select all
define service{
use generic-service
host_name ssb-voor-01
service_description JMX Memory
check_command check_nrpe!check_jmxmemory
check_interval 1
}
Please see the attached nrpe.cfg file from voor-01 the server that the plugin not works.
Thanks in advanced,
Re: NRPE: Unable to read output
Posted: Wed Mar 01, 2017 4:46 pm
by mcapra
From
ssb-voor-01_168.162.97.143, can you share the output of the following commands executed from the CLI:
Code: Select all
su nagios
/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
exit
ps aux | grep nrpe
ps aux | grep xinetd
Are you able to run the check_nrpe command against
ssb-voor-01_168.162.97.143 from the CLI of your Nagios Core machine? Like so:
Code: Select all
/path/to/plugins/check_nrpe -H 168.162.97.143 -c check_jmxmemory
Re: NRPE: Unable to read output
Posted: Thu Mar 02, 2017 6:31 am
by Katya.Ailarov
[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2940731392 | 'TotalMemory'=2940731392;3247300864;3251495168;;
root@ssb-voor-01 ~$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 7946 6002 0 06:25:42 pts/7 0:00 grep nrpe
root@ssb-voor-01 ~$ ps -ef | grep xinetd
root 7966 6002 0 06:25:58 pts/7 0:00 grep xinetd
BTW, my nagios not runing with xinetd
root@ssb-voor-01 ~$ /apps/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
root@ssb-voor-01 ~$
Re: NRPE: Unable to read output
Posted: Thu Mar 02, 2017 10:29 am
by Katya.Ailarov
[nagios@ssb-voor-01]:~ $ /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2943877120 | 'TotalMemory'=2943877120;3247300864;3251495168;;
[nagios@ssb-voor-01]:~ $ exit
logout
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep nrpe
nagios 18887 1 0 Feb 27 ? 0:00 /apps/nagios/bin/nrpe -c /apps/nagios/etc/nrpe.cfg -d
root 1761 6002 0 10:25:23 pts/7 0:00 grep nrpe
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ ps -ef | grep xinetd
root 2038 6002 0 10:26:16 pts/7 0:00 grep xinetd
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
xinetd--> my nagios not running in xinetd
from my nagios server:
-bash-4.1# hostname
ssbpops01.ssb.sungard.com
-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1#
also i see something on the remote that is not working:
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$ sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
Sorry, user root is not allowed to execute '/apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168' as nagios on ssb-voor-01.
root@ssb-voor-01 nagios-jmx-plugin-1.2.3$
on the remote host that working:
root@ssbpech01:/# sudo -u nagios /apps/nagios/libexec/nagios-jmx-plugin-1.2.3/check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.38:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2999451648 | 'TotalMemory'=2999451648;3247300864;3251495168;;
root@ssbpech01:/#
Re: NRPE: Unable to read output
Posted: Thu Mar 02, 2017 3:40 pm
by mcapra
From the Nagios Core machine, can you try executing this again:
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
Then, shortly after on 168.162.97.143, share the output of:
Code: Select all
tail -n 100 /var/log/messages | grep nrpe
Re: NRPE: Unable to read output
Posted: Sun Mar 05, 2017 7:48 am
by Katya.Ailarov
Code: Select all
-bash-4.1# /usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
NRPE: Unable to read output
-bash-4.1# hostname
ssbpops01.ssb.sungard.com
Mar 4 03:10:32 ssb-voor-01 nrpe[26586]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:32 ssb-voor-01 nrpe[26587]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:33 ssb-voor-01 nrpe[26594]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:33 ssb-voor-01 nrpe[26595]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:33 ssb-voor-01 nrpe[26596]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:38 ssb-voor-01 nrpe[26607]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:10:38 ssb-voor-01 nrpe[26612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:32 ssb-voor-01 nrpe[26859]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:32 ssb-voor-01 nrpe[26860]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:32 ssb-voor-01 nrpe[26864]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:32 ssb-voor-01 nrpe[26865]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:32 ssb-voor-01 nrpe[26872]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:38 ssb-voor-01 nrpe[26880]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:11:38 ssb-voor-01 nrpe[26881]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:32 ssb-voor-01 nrpe[27118]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:32 ssb-voor-01 nrpe[27119]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:32 ssb-voor-01 nrpe[27123]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:33 ssb-voor-01 nrpe[27131]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:33 ssb-voor-01 nrpe[27132]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:38 ssb-voor-01 nrpe[27140]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:12:38 ssb-voor-01 nrpe[27145]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:32 ssb-voor-01 nrpe[27343]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:32 ssb-voor-01 nrpe[27344]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:32 ssb-voor-01 nrpe[27345]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:32 ssb-voor-01 nrpe[27346]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:32 ssb-voor-01 nrpe[27350]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:38 ssb-voor-01 nrpe[27364]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:13:38 ssb-voor-01 nrpe[27369]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:32 ssb-voor-01 nrpe[27604]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:32 ssb-voor-01 nrpe[27612]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:32 ssb-voor-01 nrpe[27613]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:32 ssb-voor-01 nrpe[27614]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:32 ssb-voor-01 nrpe[27615]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:38 ssb-voor-01 nrpe[27625]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:14:39 ssb-voor-01 nrpe[27626]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:32 ssb-voor-01 nrpe[27833]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:32 ssb-voor-01 nrpe[27834]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:32 ssb-voor-01 nrpe[27835]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:32 ssb-voor-01 nrpe[27849]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:32 ssb-voor-01 nrpe[27850]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 4 03:15:38 ssb-voor-01 nrpe[27854]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
"syslog" 13221 lines, 1364034 characters
Mar 5 07:45:31 ssb-voor-01 nrpe[21186]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:45:31 ssb-voor-01 nrpe[21194]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:45:51 ssb-voor-01 nrpe[21200]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:45:51 ssb-voor-01 nrpe[21201]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:18 ssb-voor-01 nrpe[21396]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:23 ssb-voor-01 nrpe[21397]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:31 ssb-voor-01 nrpe[21444]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:31 ssb-voor-01 nrpe[21454]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:31 ssb-voor-01 nrpe[21461]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:31 ssb-voor-01 nrpe[21469]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:32 ssb-voor-01 nrpe[21477]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:51 ssb-voor-01 nrpe[21478]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:46:51 ssb-voor-01 nrpe[21479]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:31 ssb-voor-01 nrpe[21697]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:31 ssb-voor-01 nrpe[21698]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:31 ssb-voor-01 nrpe[21717]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:31 ssb-voor-01 nrpe[21718]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:32 ssb-voor-01 nrpe[21725]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:51 ssb-voor-01 nrpe[21735]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Mar 5 07:47:51 ssb-voor-01 nrpe[21736]: [ID 927837 mail.info] connect from ssbpops01.ssb.sungard.com
Re: NRPE: Unable to read output
Posted: Mon Mar 06, 2017 2:27 pm
by mcapra
You'll need 2 terminal sessions open for this. In the first one, run this (you may need to yum/apt install tcpdump):
Code: Select all
tcpdump -w /tmp/42740_1.pcap -i any 'port 3515'
Then, in the second session, run the check_nrpe command a few times:
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H 168.162.97.143 -c check_jmxmemory
After it's failed a few times, go back to the first session and cancel the tcpdump command. Share the resulting
/tmp/42740_1.pcap file with us to investigate.
Re: NRPE: Unable to read output
Posted: Tue Mar 07, 2017 10:15 am
by Katya.Ailarov
Hi Guys,
it will take time till the UNIX admin will install it.
meanwhile do we have any workaround?
Re: NRPE: Unable to read output
Posted: Tue Mar 07, 2017 5:03 pm
by mcapra
Not immediately unfortunately. Something is behaving outside the norm and the tcpdump will help identify roughly where things are breaking down. Looking at the raw data transmitted when the check_nrpe command is initiated will help narrow down if it's client-side or server-side.
Re: NRPE: Unable to read output
Posted: Thu Mar 09, 2017 2:50 am
by Katya.Ailarov
BTW, it is definitally the clien side (the nagios server itself) please see evidnce for that:
out put from ssbpops server -Nagios server itself:
==========================================
check-jmx of 168.162.97.143 (the server that show me NRPE:unable to read output)
==========================================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://168.162.97.143:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX CRITICAL - Error opening RMI connection: Connection refused to host: 168.162.97.143; nested exception is:
java.net.ConnectException: Connection timed out
check_jmx of 10.146.1.37 the server that the same check_jmx is working
==============================================================
-bash-4.1# ./check_jmx -U service:jmx:rmi:///jndi/rmi://10.146.1.37:7780/jmxrmi -O jboss.system:type=ServerInfo -A TotalMemory -w 3247300864 -c 3251495168
JMX OK - TotalMemory = 2982150144 | 'TotalMemory'=2982150144;3247300864;3251495168;;
please be awrae that i run this commands fron Nagios server itself
please suggest.