NRPE - strange problem

[notic]

Hi,

i have very unusual problem which i will try to explain.
NRPE is used on few servers and it's working without any problems. Yesterday i was trying to add checks for ibm db2 database and came out with this...

if i run same command as "local" nagios user on remote host.

nagios@tdb2v10:/usr/lib/nagios/plugins> ./check_database_connection --instance /home/db2inst1/ --database testdb
OK Connection to database testdb. The database is active. |'Connectable_Database'=0.9;0.6;0.3
The database is active. |'Database_Active'=0.8;0.5

BUT if i run same check (on nagios server) via NRPE or check_by_ssh, this is returned to me.

./check_nrpe -H 192.168.1.3 -c check_database_connection_testdb
A connection to database testdb was not successful. The database is active. |'Connectable_Database'=0.1;0.6;0.3
The database is active. -332|'Database_Active'=0.8;0.5

Thank you for any pointers.

[jdalrymple]

It has the makings of an SELinux or AppArmor problem, but you didn't mention your platform.

[notic]

Sorry. it's SLES 11 SP3 (s390x) on SystemZ platform.
Did check on apparmor service and it's not installed.

To me it's strange because i can run almost all checks (cpu, disks,users,...db2 instance up...etc) without any problems.
Troubles are with db2 checks which includes database paramater, but only thru NRPE, not locally (with nagios user).

[jdalrymple]

Any chance SELinux is the issue? getenforce

NRPE doesn't really do any magic. If it's not a user issue (running as nagios) and not an SELinux/AppArmor issue, then it's got to be an environment issue - meaning that you are missing part of your $PATH or some such when the process is being spun up by NRPE.

Sometimes it's useful to create an nrpe command that just runs `env` then compare it to your bash environment.

[notic]

Yes, you were right. There is a difference in PATH, RUNLEVEL and SHLVL.

NRPE env call:

TERM=xterm
NRPE_PROGRAMVERSION=2.12
LC_ALL=POSIX
REMOTE_HOST=192.168.1.11
COLUMNS=158
PATH=/sbin:/usr/sbin:/usr/local/sbin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin
RUNLEVEL=5
_=/usr/bin/env
PWD=/
DAEMON=/usr/sbin/xinetd
LANG=POSIX
PREVLEVEL=N
LINES=58
SHLVL=3
NRPE_MULTILINESUPPORT=1

local " su nagios " env:

LESSKEY=/etc/lesskey.bin
MANPATH=/usr/share/man:/usr/local/man
INFODIR=/usr/local/info:/usr/share/info:/usr/info
NNTPSERVER=news
XKEYSYMDB=/usr/share/X11/XKeysymDB
SHELL=/bin/bash
TERM=xterm
HISTSIZE=1000
PROFILEREAD=true
SSH_TTY=/dev/pts/3
USER=nagios
JRE_HOME=/usr/lib64/jvm/jre
XNLSPATH=/usr/share/X11/nls
ENV=/etc/bash.bashrc
HOSTTYPE=s390x
FROM_HEADER=
PAGER=less
CSHEDIT=emacs
XDG_CONFIG_DIRS=/etc/xdg
MINICOM=-c on
PATH=/root/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib64/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin
MAIL=/var/mail/root
CPU=s390x
JAVA_BINDIR=/usr/lib64/jvm/jre/bin
PWD=/usr/lib/nagios/plugins
INPUTRC=/etc/inputrc
JAVA_HOME=/usr/lib64/jvm/jre
LANG=POSIX
PYTHONSTARTUP=/etc/pythonstart
HOME=/home/nagios
SHLVL=2
QT_SYSTEM_DIR=/usr/share/desktop-data
OSTYPE=linux
LESS_ADVANCED_PREPROCESSOR=no
XCURSOR_THEME=DMZ
LS_OPTIONS=-N --color=tty -T 0
WINDOWMANAGER=/usr/bin/twm
LOGNAME=nagios
MACHTYPE=s390x-suse-linux
LESS=-M -I
CVS_RSH=ssh
XDG_DATA_DIRS=/usr/share
LESSOPEN=lessopen.sh %s
INFOPATH=/usr/local/info:/usr/share/info:/usr/info
LESSCLOSE=lessclose.sh %s %s
G_BROKEN_FILENAMES=1
COLORTERM=1
JAVA_ROOT=/usr/lib64/jvm/jre
OLDPWD=/root
_=/usr/bin/env

[tgriep]

Can you post your nrpe.cfg file from the remote system and a link to the check_database_connection plugin so we can look at it?
If that plugin uses java, you may need to add the full path to the java binary for the command settings in the nrpe.cfg file.

[notic]

Super.

This is my nrpe.cfg file
http://pastebin.com/E8epiTT7

and this is check_database_connection
https://github.com/angoca/monitor-db2-with-nagios/blob/master/check_database_connection

[tgriep]

In the /tmp folder on the remote system, there is a log file created when the command is run, can you post that here?
Also, where in the path is the db2 program installed? You may need to add that path to the default path so when the command it run, it will be available to the nagios user.

[notic]

nagios@db2v10:/usr/lib/nagios/plugins> cat /tmp/check_database_connection.log
2015-11-04-08.17.59 28347 Started check_database_connection -i /home/db2inst1/ -d testdb
2015-11-04-08.18.01 28347 Ended check_database_connection -i /home/db2inst1/ -d testdb

by default path i didn't fully understand you, what i did you mean. I also tried to add nagios user to root and db2 "super group", but i always get same "error".

[jdalrymple]

I'm not sure if it matters, however your command syntax from nrpe.cfg:

Code: Select all

command[check_check_connection_qty]=/usr/lib/nagios/plugins/check_connection_qty -i /home/db2inst1/ -d testdb

is not identical to the syntax used in your earlier demonstrated success:

Code: Select all

nagios@tdb2v10:/usr/lib/nagios/plugins> ./check_database_connection --instance /home/db2inst1/ --database testdb
OK Connection to database testdb. The database is active. |'Connectable_Database'=0.9;0.6;0.3
The database is active. |'Database_Active'=0.8;0.5