Wrong output showing in nagios console

[informatica]

Hi Team,

we have written one shell script which to check the cluster status and provide the output. The script is working in agent where as when we are calling from nagios we are not able to get the valid output. Please find the below details.


[root@asvrh libexec]# cat check_rmq_cluster.sh
#!/bin/bash
source /opt/morpheus/embedded/rabbitmq/.profile
node_count=`/opt/morpheus/embedded/bin/rabbitmqctl cluster_status | grep "Running Nodes" -A 5 | grep @ | wc -l`
echo $node_count

case $node_count in
3)
echo "OK: RabbitMQ is clustered on 3 nodes"
exit 0;;
2)
echo "WARN: RabbitMQ is only clustered on 2 nodes"
exit 2;;
1)
echo "CRITICAL: RabbitMQ is not clustered"
exit 1;;
*) ;;
esac

if we execute the script in server it self we are getting the valid output .
[root@asvrhmo libexec]# ./check_rmq_cluster.sh
3
OK: RabbitMQ is clustered on 3 nodes
[root@asvrhmorp libexec]# cat /usr/local/nagios/etc/nrpe/custom.cfg
command[check_es_cluster]=/usr/local/nagios/libexec/check_es_cluster.sh
command[check_rmq_cluster]=/usr/local/nagios/libexec/check_rmq_cluster.sh
command[check_server_servicestatus]=/usr/local/nagios/libexec/check-server_servicestatus.sh
command[elasticsearch_servicestatus]=/usr/local/nagios/libexec/elasticsearch_servicestatus.sh
command[guacd_servicestatus]=/usr/local/nagios/libexec/guacd_servicestatus.sh
command[morpheus_ui_servicestatus]=/usr/local/nagios/libexec/morpheus-ui_servicestatus.sh
command[nginx_servicestatus]=/usr/local/nagios/libexec/nginx_servicestatus.sh
command[rabbit_servicestatus]=/usr/local/nagios/libexec/rabbit_servicestatus.sh

when we are calling from nagios i am not getting valid output

[nagios@qy root]$ /usr/local/nagios/libexec/check_nrpe -2 -H XXXX -t 60 -u -c check_rmq_cluster
declare -x ERL_EPMD_ADDRESS="0.0.0.0"
declare -x EXTRAOPTIONS=""
declare -x HOME="/opt/morpheus/embedded/rabbitmq"
declare -x LANG="en_US.UTF-8"
declare -x LC_ALL="en_US.UTF-8"
declare -x NRPE_MULTILINESUPPORT="1"
declare -x NRPE_PROGRAMVERSION="2.15"
declare -x OLDPWD
declare -x PATH="/opt/morpheus/bin:/opt/morpheus/embedded/bin:/opt/morpheus/embedded/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
declare -x PWD="/"
declare -x RABBITMQ_CONF_ENV_FILE="/opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf"
declare -x REMOTE_HOST="::ffff:10.25.32.13"
declare -x SHLVL="2"
declare -x XINETD_LANG="en_US"
0

Could you please help us to resolve the issue.

[dchurch]

[root@asvrh libexec]# cat check_rmq_cluster.sh
[root@asvrhmo libexec]# ./check_rmq_cluster.sh
[root@asvrhmorp libexec]# cat /usr/local/nagios/etc/nrpe/custom.cfg

Is this a mistake? Are these prompts from the same host, or are they three different hosts?

If it is a mistake, and they're all the same computer, you could try changing the line to:

Code: Select all

command[check_rmq_cluster]=bash /usr/local/nagios/libexec/check_rmq_cluster.sh

[informatica]

as suggested we did same but same issue was occured.

[root@asvrhmorphapp1 nrpe]# cat custom.cfg
command[check_es_cluster]=/usr/local/nagios/libexec/check_es_cluster.sh
command[check_rmq_cluster]=bash /usr/local/nagios/libexec/check_rmq_cluster.sh

[root@asvrhmorphapp1 nrpe]# bash /usr/local/nagios/libexec/check_rmq_cluster.sh
3
OK: RabbitMQ is clustered on 3 nodes
[root@asvrhmorphapp1 nrpe]#

[nagios@qy-nagios-a root]$ /usr/local/nagios/libexec/check_nrpe -2 -H asvrhmorphapp1 -t 60 -u -c check_rmq_cluster
declare -x ERL_EPMD_ADDRESS="0.0.0.0"
declare -x EXTRAOPTIONS=""
declare -x HOME="/opt/morpheus/embedded/rabbitmq"
declare -x LANG="en_US.UTF-8"
declare -x LC_ALL="en_US.UTF-8"
declare -x NRPE_MULTILINESUPPORT="1"
declare -x NRPE_PROGRAMVERSION="2.15"
declare -x OLDPWD
declare -x PATH="/opt/morpheus/bin:/opt/morpheus/embedded/bin:/opt/morpheus/embedded/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
declare -x PWD="/"
declare -x RABBITMQ_CONF_ENV_FILE="/opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf"
declare -x REMOTE_HOST="::ffff:10.25.32.13"
declare -x SHLVL="2"
declare -x XINETD_LANG="en_US"
0


All are same computer, we just removed hostname as security reason.

[dchurch]

It's running thru xinetd. Can you inspect your xinetd configuration to see if it's running nrpe with any funny options?

What's the output from the following command (on the NRPE machine)?

Code: Select all

ps -A x -f |grep -i nrpe

[informatica]

Hi Team,

Please find the below output.

[root@asvrhmorphapp1 ~]# service xinetd status
Redirecting to /bin/systemctl status xinetd.service
● xinetd.service - Xinetd A Powerful Replacement For Inetd
Loaded: loaded (/usr/lib/systemd/system/xinetd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-03-23 11:18:56 CDT; 23h ago
Process: 32260 ExecStart=/usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid $EXTRAOPTIONS (code=exited, status=0/SUCCESS)
Main PID: 32261 (xinetd)
Tasks: 1
CGroup: /system.slice/xinetd.service
└─32261 /usr/sbin/xinetd -stayalive -pidfile /var/run/xinetd.pid

Mar 24 10:56:46 asvrhmorphapp1.informatica.com xinetd[32261]: START: nrpe pid=20706 from=::ffff:10.25.32.13
Mar 24 10:56:46 asvrhmorphapp1.informatica.com xinetd[32261]: EXIT: nrpe status=0 pid=20706 duration=0(sec)
Mar 24 10:57:22 asvrhmorphapp1.informatica.com xinetd[32261]: START: nrpe pid=20805 from=::ffff:10.25.32.13
Mar 24 10:57:23 asvrhmorphapp1.informatica.com xinetd[32261]: EXIT: nrpe status=0 pid=20805 duration=1(sec)
Mar 24 10:57:32 asvrhmorphapp1.informatica.com xinetd[32261]: START: nrpe pid=20832 from=::ffff:10.25.32.13
Mar 24 10:57:32 asvrhmorphapp1.informatica.com xinetd[32261]: EXIT: nrpe status=0 pid=20832 duration=0(sec)
Mar 24 10:57:55 asvrhmorphapp1.informatica.com xinetd[32261]: START: nrpe pid=20907 from=::ffff:10.25.32.13
Mar 24 10:58:03 asvrhmorphapp1.informatica.com xinetd[32261]: EXIT: nrpe status=0 pid=20907 duration=8(sec)
Mar 24 10:58:20 asvrhmorphapp1.informatica.com xinetd[32261]: START: nrpe pid=21310 from=::ffff:10.25.32.13
Mar 24 10:58:20 asvrhmorphapp1.informatica.com xinetd[32261]: EXIT: nrpe status=0 pid=21310 duration=0(sec)
[root@asvrhmorphapp1 ~]# ps -A x -f |grep -i nrpe
root 21359 20906 0 10:58 pts/1 S+ 0:00 grep --color=auto -i nrpe
[root@asvrhmorphapp1 ~]# ps -A x -f |grep -i nrpe
root 21359 20906 0 10:58 pts/1 S+ 0:00 grep --color=auto -i nrpe
[root@asvrhmorphapp1 ~]# ps -ef | grep nrpe
root 21504 20906 0 10:59 pts/1 00:00:00 grep --color=auto nrpe
[root@asvrhmorphapp1 ~]# ps -ef | grep 5666
root 21556 20906 0 10:59 pts/1 00:00:00 grep --color=auto 5666

[ssax]

Those declares are coming from this:

Code: Select all

source /opt/morpheus/embedded/rabbitmq/.profile

Please attach (or PM) that file.

[ssax]

Try changing this in your script:

Code: Select all

node_count=`/opt/morpheus/embedded/bin/rabbitmqctl cluster_status | grep "Running Nodes" -A 5 | grep @ | wc -l`

To this:

Code: Select all

node_count=`/opt/morpheus/embedded/bin/rabbitmqctl cluster_status 2>&1`

Then run the check again and send the full output. You can revert the change afterwards.

PM me this file as well:

Code: Select all

/opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf

[informatica]

We replied as PM please check

[ssax]

The nagios user probably doesn't have permissions.

Please run these commands and send the output (su - to nagios first):

Code: Select all

su - nagios
bash -x /usr/local/nagios/libexec/check_rmq_cluster.sh

[informatica]

Hi Team,

Please find the below logs.

nagios@asvrhmorphapp1 libexec]$ ./check_rmq_cluster.sh
/opt/morpheus/embedded/rabbitmq/.profile: line 7: /opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf: Permission denied
cut: /opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf: Permission denied
declare -x ERL_EPMD_ADDRESS="0.0.0.0"
declare -x GROUP="users"
declare -x HOME="/opt/morpheus/embedded/rabbitmq"
declare -x HOST="asvrhmorphapp1.informatica.com"
declare -x HOSTNAME="asvrhmorphapp1.informatica.com"
declare -x HOSTTYPE="x86_64-linux"
declare -x LANG="en_US.UTF-8"
declare -x LC_ALL="en_US.UTF-8"
declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s"
declare -x LOGNAME="nagios"
declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:"
declare -x MACHTYPE="x86_64"
declare -x MAIL="/var/spool/mail/nagios"
declare -x OLDPWD
declare -x OSTYPE="linux"
declare -x PATH="/opt/morpheus/bin:/opt/morpheus/embedded/bin:/opt/morpheus/embedded/sbin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin"
declare -x PWD="/usr/local/nagios/libexec"
declare -x RABBITMQ_CONF_ENV_FILE="/opt/morpheus/embedded/rabbitmq/etc/rabbitmq-env.conf"
declare -x REMOTEHOST="10.65.53.30"
declare -x SHELL="/bin/csh"
declare -x SHLVL="2"
declare -x TERM="xterm"
declare -x USER="nagios"
declare -x VENDOR="unknown"
declare -x XDG_SESSION_ID="30008"

11:29:46.247 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:47.218 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:47.220 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:48.167 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:48.168 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:49.133 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:49.134 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:50.121 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:50.123 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:51.149 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces

11:29:51.151 [error] Error when reading /opt/morpheus/embedded/rabbitmq/.erlang.cookie: eacces