jstat is failing to show a proper result

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
runzelpunzel
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

jstat is failing to show a proper result

Post by runzelpunzel »

Hi @ll,

I am trying to gather some stats from my JVM, here: a tomcat process.

I've been trying for ages to get my head around a problem but did not succeed, thus I'm seeking for your help.
I think I have read almost each and every topic regarding this issue, yet the issue is still not sorted out.

I startet with the script "check_jstat.sh" from here: https://exchange.nagios.org/directory/P ... at/details

I think I've included all mandatory changes to the script (e.g. call "jstat" using sudo) and provisioned access to the script in /etc/sudoers.
I have also set up

Code: Select all

Defaults:nagios !requiretty
The NRPE-execution is failing with Can't get GC statistics
wheres the manual call of the script (as NRPE-user 'nagios') returns the expected result:

Code: Select all

nagios@MYMACHINE[~]> sudo /usr/lib/nagios/plugins/check_jstat.sh -p 1314 -w 90 -c 95
OK:jstat process 1314 alive, | 1314_heap=1991275520B;3865470566;4080218931;0;4294967296 1314_heap_ratio=46%;90;95;0;100 1314_perm=573860864B;1481637888;1563951104;0;1646264320 1314_perm_ratio=34%;90;95;0;100
The failing command is the call of 'jstat' (gc=$(sudo jstat -gc $pid | tail -1 | sed -e 's/[ ][ ]*/ /g')), thus I have extracted the failing command to a very simple script, which reads as follows:

Code: Select all

cat /usr/lib/nagios/plugins/check_test.sh

#!/bin/bash
sudo jstat -gc 1314 | tail -1 | sed -e 's/[ ][ ]*/ /g' > ~/check_test.log 2>&1
exit 5
The result is, what you would expect:

Code: Select all

nagios@MYMACHINE[~]> sudo /usr/lib/nagios/plugins/check_test.sh
root@MYMACHINE[/usr/lib/nagios/plugins]> cat ~root/check_test.log
38912,0 39424,0 23936,3 0,0 1318912,0 672096,7 1664512,0 1338755,6 646840,0 560757,3 88288,0 71962,9 29418 2113,383 110 110,260 2223,643
The nrpe.cfg@MYMACHINE contains the call as follows:

Code: Select all

command[check_test]=sudo /usr/lib/nagios/plugins/check_test.sh
Calling this command from my Nagios-server as follows

Code: Select all

nagios@nagios[/usr/local/nagios/libexec]> ./check_nrpe -H p083.persis.de -c check_test
leads to an error NRPE: Unable to read output

The debug log show:

Code: Select all

Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x port 12960
Feb  4 15:44:53 p083 nrpe[16032]: Host address is in allowed_hosts
Feb  4 15:44:53 p083 nrpe[16032]: Host x.x.x.x is asking for command 'check_test' to be run...
Feb  4 15:44:53 p083 nrpe[16032]: Running command: sudo /usr/lib/nagios/plugins/check_test.sh
Feb  4 15:44:53 p083 nrpe[16032]: Command completed with return code 3 and output:
Feb  4 15:44:53 p083 nrpe[16032]: Return Code: 3, Output: NRPE: Unable to read output
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x closed.
The check_test.log is not being written into.

What am I doing wrong?

Any help is greaty appreciated!

Best regards,
Dennis
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: jstat is failing to show a proper result

Post by Box293 »

You could try removing sudo from the command directive in NRPE and instead define command_prefix=/usr/bin/sudo in nrpe.cfg.

Alternatively you can turn on NRPE debugging to collect more information.

Edit the file:
/usr/local/nagios/etc/nrpe.cfg

Define
debug=1
(it will currently be debug=0)

Save the file and
service xinetd restart

Now we need to add an option to the rsyslog server so it processes debug messages
Edit the file:
/etc/rsyslogd.conf
Find /var/log/messages
The line in the config file will look like:
*.info;mail.none;authpriv.none;cron.none /var/log/messages

We need to add the following to the line:
*.info;mail.none;authpriv.none;cron.none;daemon.debug /var/log/messages

Save the file and
service rsyslog restart
Now there should be more information logged in /var/log/messages
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
runzelpunzel
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Post by runzelpunzel »

Hi @Box293,

thanks for looking into this and for your feedback.

I honestly don't want to add a command_prefix for sudo in general in nrpe.cfg as it will cause *every* check-command to be executed via sudo, which is not what I want.

debug=1 has already been defined and debug output is being written to /var/log/daemon.log, see my original post, as per the configuration of /etc/rsyslog.conf:

Code: Select all

daemon.*                        -/var/log/daemon.log
Thus, the debug-output is very basic, e.g.

Code: Select all

Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x port 12960
Feb  4 15:44:53 p083 nrpe[16032]: Host address is in allowed_hosts
Feb  4 15:44:53 p083 nrpe[16032]: Host x.x.x.x is asking for command 'check_test' to be run...
Feb  4 15:44:53 p083 nrpe[16032]: Running command: sudo /usr/lib/nagios/plugins/check_test.sh
Feb  4 15:44:53 p083 nrpe[16032]: Command completed with return code 3 and output:
Feb  4 15:44:53 p083 nrpe[16032]: Return Code: 3, Output: NRPE: Unable to read output
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x closed.
I couldn't find a way to increase the verbosity of the debugmode.

Any further ideas?

Many thanks,
Dennis
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: jstat is failing to show a proper result

Post by Box293 »

Maybe your command needs the path to sudo specifically defined like /usr/bin/sudo as it may not be in the environment path when NRPE runs.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
runzelpunzel
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Post by runzelpunzel »

Unfortunately, this is all not the problem. The thing is, that jstat -gc is failing to work via NRPE, whereas it is working when being called locally on MYMACHINE.

To clarifiy things, pls. take a look at the following test-cases:
Extract of nrpe.cfg on MYMACHINE (x.x.x.x):

Code: Select all

command[check_sudo_test1]=sudo whoami
command[check_sudo_test2]=sudo env
command[check_sudo_test3]=sudo jstat -help
command[check_sudo_test4]=sudo jstat -gc 1314
Executed tests and results from Nagios-server:

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test1
root

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test2
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAIL=/var/mail/root
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
TERM=unknown
SUDO_COMMAND=/usr/bin/env
SUDO_USER=nagios
SUDO_UID=115
SUDO_GID=122

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test3
Usage: jstat -help|-options
       jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]

Definitions:
  <option>      An option reported by the -options option
  <vmid>        Virtual Machine Identifier. A vmid takes the following form:
                     <lvmid>[@<hostname>[:<port>]]
                Where <lvmid> is the local vm identifier for the target
                Java virtual machine, typically a process id; <hostname> is
                the name of the host running the target Java virtual machine;
                and <port> is the port number for the rmiregistry on the
                target host. See the jvmstat documentation for a more complete
                description of the Virtual Machine Identifier.
  <lines>       Number of samples between header lines.
  <interval>    Sampling interval. The following forms are allowed:
                    <n>["ms"|"s"]
                Where <n> is an integer and the suffix specifies the units as
                milliseconds("ms") or seconds("

Code: Select all

/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test4
NRPE: Unable to read output
Whereas when I run the command sudo jstat -gc 1314 locally on MYMACHINE as nagios using sudo I get the correct result:

Code: Select all

nagios@MYMACHINE[~]> sudo jstat -gc 1314
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
12288,0 22016,0 12145,1  0,0   690688,0 356714,0 1654272,0  1424211,3  646840,0 559281,5 88288,0 71507,4  30964 2236,129  115   115,107 2351,235
So that doesn't make too much sense to me... :-/
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: jstat is failing to show a proper result

Post by Box293 »

Box293 wrote:Maybe your command needs the path to sudo specifically defined like /usr/bin/sudo as it may not be in the environment path when NRPE runs.
Did you actually try what I suggested?

We're troubleshooting an issue and if you don't try the troubleshooting steps provided then it's hard to rule things out.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
runzelpunzel
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Post by runzelpunzel »

Hi @box293,

thanks for your reply.

I did not provide that information as - to the best of my knowledge - I thought this was clear as per the results of the tests that I provided, i.e. sudo whoami as well as sudo env.
If NRPE wouldn't find sudo in the path, whoami would return nagios instead of root.

Nevertheless I reconfigured the command to read command[check_sudo_test4]=/usr/bin/sudo jstat -gc 1314.
The result remains yet the same.

Any more ideas?
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: jstat is failing to show a proper result

Post by Box293 »

I would also provide the exact location of jstat in the command, like:

command[check_sudo_test4]=/usr/bin/sudo /the/location/here/jstat -gc 1314
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
runzelpunzel
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Post by runzelpunzel »

same result.

Any more ideas?

Best regards,
Dennis

PS: Even though I have subsribed the entire topic I do not get any mail notification on your reply, thus the delay in my response.
User avatar
Box293
Too Basu
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia
Contact:

Re: jstat is failing to show a proper result

Post by Box293 »

In your nrpe.cfg what do you have defined for nrpe_user and nrpe_group ?

As for forum notifications, check the user control panel to see if you have enabled them.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
Locked