jstat is failing to show a proper result

Engage with the community of users including those using the open source solutions.
Includes Nagios Core, Plugins, and NCPA

jstat is failing to show a proper result

Postby runzelpunzel » Tue Feb 04, 2020 9:50 am

Hi @ll,

I am trying to gather some stats from my JVM, here: a tomcat process.

I've been trying for ages to get my head around a problem but did not succeed, thus I'm seeking for your help.
I think I have read almost each and every topic regarding this issue, yet the issue is still not sorted out.

I startet with the script "check_jstat.sh" from here: https://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jstat/details

I think I've included all mandatory changes to the script (e.g. call "jstat" using sudo) and provisioned access to the script in /etc/sudoers.
I have also set up
Code: Select all
Defaults:nagios !requiretty


The NRPE-execution is failing with Can't get GC statistics
wheres the manual call of the script (as NRPE-user 'nagios') returns the expected result:
Code: Select all
nagios@MYMACHINE[~]> sudo /usr/lib/nagios/plugins/check_jstat.sh -p 1314 -w 90 -c 95
OK:jstat process 1314 alive, | 1314_heap=1991275520B;3865470566;4080218931;0;4294967296 1314_heap_ratio=46%;90;95;0;100 1314_perm=573860864B;1481637888;1563951104;0;1646264320 1314_perm_ratio=34%;90;95;0;100


The failing command is the call of 'jstat' (gc=$(sudo jstat -gc $pid | tail -1 | sed -e 's/[ ][ ]*/ /g')), thus I have extracted the failing command to a very simple script, which reads as follows:
Code: Select all
cat /usr/lib/nagios/plugins/check_test.sh

#!/bin/bash
sudo jstat -gc 1314 | tail -1 | sed -e 's/[ ][ ]*/ /g' > ~/check_test.log 2>&1
exit 5


The result is, what you would expect:
Code: Select all
nagios@MYMACHINE[~]> sudo /usr/lib/nagios/plugins/check_test.sh
root@MYMACHINE[/usr/lib/nagios/plugins]> cat ~root/check_test.log
38912,0 39424,0 23936,3 0,0 1318912,0 672096,7 1664512,0 1338755,6 646840,0 560757,3 88288,0 71962,9 29418 2113,383 110 110,260 2223,643


The nrpe.cfg@MYMACHINE contains the call as follows:
Code: Select all
command[check_test]=sudo /usr/lib/nagios/plugins/check_test.sh


Calling this command from my Nagios-server as follows
Code: Select all
nagios@nagios[/usr/local/nagios/libexec]> ./check_nrpe -H p083.persis.de -c check_test

leads to an error NRPE: Unable to read output

The debug log show:
Code: Select all
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x port 12960
Feb  4 15:44:53 p083 nrpe[16032]: Host address is in allowed_hosts
Feb  4 15:44:53 p083 nrpe[16032]: Host x.x.x.x is asking for command 'check_test' to be run...
Feb  4 15:44:53 p083 nrpe[16032]: Running command: sudo /usr/lib/nagios/plugins/check_test.sh
Feb  4 15:44:53 p083 nrpe[16032]: Command completed with return code 3 and output:
Feb  4 15:44:53 p083 nrpe[16032]: Return Code: 3, Output: NRPE: Unable to read output
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x closed.


The check_test.log is not being written into.

What am I doing wrong?

Any help is greaty appreciated!

Best regards,
Dennis
runzelpunzel
 
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Postby Box293 » Tue Feb 04, 2020 7:45 pm

You could try removing sudo from the command directive in NRPE and instead define command_prefix=/usr/bin/sudo in nrpe.cfg.

Alternatively you can turn on NRPE debugging to collect more information.

Edit the file:
/usr/local/nagios/etc/nrpe.cfg

Define
debug=1
(it will currently be debug=0)

Save the file and
service xinetd restart

Now we need to add an option to the rsyslog server so it processes debug messages
Edit the file:
/etc/rsyslogd.conf
Find /var/log/messages
The line in the config file will look like:
*.info;mail.none;authpriv.none;cron.none /var/log/messages

We need to add the following to the line:
*.info;mail.none;authpriv.none;cron.none;daemon.debug /var/log/messages

Save the file and
service rsyslog restart
Now there should be more information logged in /var/log/messages
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
Box293
Too Basu
 
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia

Re: jstat is failing to show a proper result

Postby runzelpunzel » Wed Feb 05, 2020 4:45 am

Hi @Box293,

thanks for looking into this and for your feedback.

I honestly don't want to add a command_prefix for sudo in general in nrpe.cfg as it will cause *every* check-command to be executed via sudo, which is not what I want.

debug=1 has already been defined and debug output is being written to /var/log/daemon.log, see my original post, as per the configuration of /etc/rsyslog.conf:
Code: Select all
daemon.*                        -/var/log/daemon.log


Thus, the debug-output is very basic, e.g.
Code: Select all
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x port 12960
Feb  4 15:44:53 p083 nrpe[16032]: Host address is in allowed_hosts
Feb  4 15:44:53 p083 nrpe[16032]: Host x.x.x.x is asking for command 'check_test' to be run...
Feb  4 15:44:53 p083 nrpe[16032]: Running command: sudo /usr/lib/nagios/plugins/check_test.sh
Feb  4 15:44:53 p083 nrpe[16032]: Command completed with return code 3 and output:
Feb  4 15:44:53 p083 nrpe[16032]: Return Code: 3, Output: NRPE: Unable to read output
Feb  4 15:44:53 p083 nrpe[16032]: Connection from x.x.x.x closed.


I couldn't find a way to increase the verbosity of the debugmode.

Any further ideas?

Many thanks,
Dennis
runzelpunzel
 
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Postby Box293 » Wed Feb 05, 2020 5:22 pm

Maybe your command needs the path to sudo specifically defined like /usr/bin/sudo as it may not be in the environment path when NRPE runs.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
Box293
Too Basu
 
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia

Re: jstat is failing to show a proper result

Postby runzelpunzel » Thu Feb 06, 2020 2:29 am

Unfortunately, this is all not the problem. The thing is, that jstat -gc is failing to work via NRPE, whereas it is working when being called locally on MYMACHINE.

To clarifiy things, pls. take a look at the following test-cases:
Extract of nrpe.cfg on MYMACHINE (x.x.x.x):
Code: Select all
command[check_sudo_test1]=sudo whoami
command[check_sudo_test2]=sudo env
command[check_sudo_test3]=sudo jstat -help
command[check_sudo_test4]=sudo jstat -gc 1314


Executed tests and results from Nagios-server:
Code: Select all
/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test1
root

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test2
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
MAIL=/var/mail/root
LOGNAME=root
USER=root
HOME=/root
SHELL=/bin/bash
TERM=unknown
SUDO_COMMAND=/usr/bin/env
SUDO_USER=nagios
SUDO_UID=115
SUDO_GID=122

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test3
Usage: jstat -help|-options
       jstat -<option> [-t] [-h<lines>] <vmid> [<interval> [<count>]]

Definitions:
  <option>      An option reported by the -options option
  <vmid>        Virtual Machine Identifier. A vmid takes the following form:
                     <lvmid>[@<hostname>[:<port>]]
                Where <lvmid> is the local vm identifier for the target
                Java virtual machine, typically a process id; <hostname> is
                the name of the host running the target Java virtual machine;
                and <port> is the port number for the rmiregistry on the
                target host. See the jvmstat documentation for a more complete
                description of the Virtual Machine Identifier.
  <lines>       Number of samples between header lines.
  <interval>    Sampling interval. The following forms are allowed:
                    <n>["ms"|"s"]
                Where <n> is an integer and the suffix specifies the units as
                milliseconds("ms") or seconds("

Code: Select all
/usr/local/nagios/libexec/check_nrpe -H x.x.x.x -c check_sudo_test4
NRPE: Unable to read output


Whereas when I run the command sudo jstat -gc 1314 locally on MYMACHINE as nagios using sudo I get the correct result:
Code: Select all
nagios@MYMACHINE[~]> sudo jstat -gc 1314
S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT     GCT
12288,0 22016,0 12145,1  0,0   690688,0 356714,0 1654272,0  1424211,3  646840,0 559281,5 88288,0 71507,4  30964 2236,129  115   115,107 2351,235


So that doesn't make too much sense to me... :-/
runzelpunzel
 
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Postby Box293 » Thu Feb 06, 2020 5:14 pm

Box293 wrote:Maybe your command needs the path to sudo specifically defined like /usr/bin/sudo as it may not be in the environment path when NRPE runs.


Did you actually try what I suggested?

We're troubleshooting an issue and if you don't try the troubleshooting steps provided then it's hard to rule things out.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
Box293
Too Basu
 
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia

Re: jstat is failing to show a proper result

Postby runzelpunzel » Fri Feb 07, 2020 2:01 am

Hi @box293,

thanks for your reply.

I did not provide that information as - to the best of my knowledge - I thought this was clear as per the results of the tests that I provided, i.e. sudo whoami as well as sudo env.
If NRPE wouldn't find sudo in the path, whoami would return nagios instead of root.

Nevertheless I reconfigured the command to read command[check_sudo_test4]=/usr/bin/sudo jstat -gc 1314.
The result remains yet the same.

Any more ideas?
runzelpunzel
 
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Postby Box293 » Mon Feb 10, 2020 4:40 pm

I would also provide the exact location of jstat in the command, like:

command[check_sudo_test4]=/usr/bin/sudo /the/location/here/jstat -gc 1314
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
Box293
Too Basu
 
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia

Re: jstat is failing to show a proper result

Postby runzelpunzel » Thu Feb 13, 2020 2:08 am

same result.

Any more ideas?

Best regards,
Dennis

PS: Even though I have subsribed the entire topic I do not get any mail notification on your reply, thus the delay in my response.
runzelpunzel
 
Posts: 16
Joined: Thu Jul 05, 2018 2:14 am

Re: jstat is failing to show a proper result

Postby Box293 » Thu Feb 13, 2020 5:16 pm

In your nrpe.cfg what do you have defined for nrpe_user and nrpe_group ?

As for forum notifications, check the user control panel to see if you have enabled them.
As of May 25th, 2018, all communications with Nagios Enterprises and its employees are covered under our new Privacy Policy.
User avatar
Box293
Too Basu
 
Posts: 5126
Joined: Sun Feb 07, 2010 10:55 pm
Location: Deniliquin, Australia

Next

Return to Community Support

Who is online

Users browsing this forum: Google [Bot] and 31 guests