NRPE: Command 'check_init_service' not defined

[linuser]

I am able to get this to work from the remote system but not the nagios server. The error in the subject line is what I am getting. From the remote system I can run this from command line:

Code: Select all

/usr/lib64/nagios/plugins/check_init_service bgpd 

and get this:

Code: Select all

Redirecting to /bin/systemctl status  bgpd.service
bgpd.service - BGP routing daemon
   Loaded: loaded (/usr/lib/systemd/system/bgpd.service; enabled)
   Active: active (running) since Tue 2015-10-27 10:40:44 CDT; 2h 18min ago
  Process: 2081 ExecStart=/usr/sbin/bgpd -u quagga -g quagga -f /etc/quagga/bgpd.conf -d -P 17500 (code=exited, status=0/SUCCESS)
 Main PID: 2267 (bgpd)
   CGroup: /system.slice/bgpd.service
           └─2267 /usr/sbin/bgpd -u quagga -g quagga -f /etc/quagga/bgpd.conf -d -P 17500
 

So far so good. Here are the command arguments from the nrpe.cfg file on the remote server.

Code: Select all

command[check_init1]=/usr/lib64/nagios/plugins/check_init_service bgpd
command[check_init2]=/usr/lib64/nagios/plugins/check_init_service zebra
 

However when I go to set all this up on the nagios server to check it remotely it fails. From the command line I run:

Code: Select all

./check_nrpe -n -H 192.168.255.70 -t 30 -c check_init_service bgpd 

I have tried this with single quotes around bgpd, double quotes, no quotes, appending -a before the argument. Nothing works and it returns:

Code: Select all

 NRPE: Command 'check_init_service' not defined

Problem is- I am not sure what its wanting here. There is also nothing helpful in any logs. My nagios server config is setup as follows:

from commands.cfg:

Code: Select all

# 'check_init_service' command definition
define command{
        command_name    check_init_service
        command_line    $USER1$/check_nrpe -n -H $HOSTADDRESS$ -t 30 -c check_init_service -a $ARG1$
        }
 

and In the host file config:

Code: Select all

define service{
        use                             unix-service         ; Name of service template to use
        host_name                       <my_domain>
        service_description             Quagga-bgpd
        check_command                   check_nrpe-2.24!check_init1
        notification_period             workhours
        notifications_enabled           1
        }

define service{
        use                             unix-service         ; Name of service template to use
        host_name                       <my_domain>
        service_description             Quagga-zebra
        check_command                   check_nrpe-2.24!check_init2
        notification_period             workhours
        notifications_enabled           1
        }
 

I think if I can get this to work from command line with nrpe It may work in nagios. But I am completely stuck at this point and not sure what to do from here. Any help is greatly appreciated.

[hsmith]

Hate to ask the obvious, but did you restart xinetd after editing the configuration on the remote server?

[linuser]

Hate to ask the obvious, but did you restart xinetd after editing the configuration on the remote server?

No, I'm not using the config that way. Nrpe runs as a standalone daemon, and nrpe listens on 5666, not xinetd.

Thanks!

[hsmith]

All right, wasn't sure which way you had it. At any rate, was it restarted so the new config can take effect?

[linuser]

All right, wasn't sure which way you had it. At any rate, was it restarted so the new config can take effect?

Yes, many times.

[rkennedy]

On your nrpe.cfg on the remote server, you have defined the commands as -

Code: Select all

command[check_init1]=/usr/lib64/nagios/plugins/check_init_service bgpd
command[check_init2]=/usr/lib64/nagios/plugins/check_init_service zebra

When you reference these from your command on the Nagios server you need to use those as your -c variable.

Example -

Code: Select all

./check_nrpe -n -H 192.168.255.70 -t 30 -c check_init1

Can you let me know if that works for you or not?

[linuser]

On your nrpe.cfg on the remote server, you have defined the commands as -

Code: Select all

command[check_init1]=/usr/lib64/nagios/plugins/check_init_service bgpd
command[check_init2]=/usr/lib64/nagios/plugins/check_init_service zebra

When you reference these from your command on the Nagios server you need to use those as your -c variable.

Example -

Code: Select all

./check_nrpe -n -H 192.168.255.70 -t 30 -c check_init1

Can you let me know if that works for you or not?

Yes, that better. But it still does not work. Now I get:

Code: Select all

NRPE: Unable to read output 

As a test I ran another command for NTP which uses "check_time1" as its listed in the cfg file. That worked and returned output.

On the server in /var/log/nagios/nagios.log I see this:

Code: Select all

[1445980219] SERVICE NOTIFICATION: nagios;<hostname>;Quagga-zebra;CRITICAL;notify-service-by-email;NRPE: Unable to read output 

And in /var/log/audit/audit.log this pops in every time I run the command:

Code: Select all

 type=AVC msg=audit(1445982801.776:1442): avc:  denied  { search } for  pid=6120 comm="mountpoint" name="/" dev="tmpfs" ino=11272 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:cgroup_t:s0 tclass=dir
type=SYSCALL msg=audit(1445982801.776:1442): arch=c000003e syscall=4 success=no exit=-13 a0=7fffee895f12 a1=7fffee893bb0 a2=7fffee893bb0 a3=7fffee893920 items=0 ppid=6118 pid=6120 auid=4294967295 uid=997 gid=995 euid=997 suid=997 fsuid=997 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295 comm="mountpoint" exe="/usr/bin/mountpoint" subj=system_u:system_r:nrpe_t:s0 key=(null)
type=AVC msg=audit(1445982801.783:1443): avc:  denied  { execute } for  pid=6117 comm="service" name="systemctl" dev="dm-1" ino=2101040 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1445982801.783:1443): arch=c000003e syscall=59 success=no exit=-13 a0=1f17280 a1=1f409d0 a2=1f18a90 a3=7fffbdd79f30 items=0 ppid=6115 pid=6117 auid=4294967295 uid=997 gid=995 euid=997 suid=997 fsuid=997 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295 comm="service" exe="/usr/bin/bash" subj=system_u:system_r:nrpe_t:s0 key=(null)
type=AVC msg=audit(1445982801.783:1444): avc:  denied  { getattr } for  pid=6117 comm="service" path="/usr/bin/systemctl" dev="dm-1" ino=2101040 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1445982801.783:1444): arch=c000003e syscall=4 success=no exit=-13 a0=1f17280 a1=7fffbdd7a100 a2=7fffbdd7a100 a3=7fffbdd79f30 items=0 ppid=6115 pid=6117 auid=4294967295 uid=997 gid=995 euid=997 suid=997 fsuid=997 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295 comm="service" exe="/usr/bin/bash" subj=system_u:system_r:nrpe_t:s0 key=(null)
type=AVC msg=audit(1445982801.783:1445): avc:  denied  { getattr } for  pid=6117 comm="service" path="/usr/bin/systemctl" dev="dm-1" ino=2101040 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1445982801.783:1445): arch=c000003e syscall=4 success=no exit=-13 a0=1f17280 a1=7fffbdd7a0e0 a2=7fffbdd7a0e0 a3=7fffbdd79f30 items=0 ppid=6115 pid=6117 auid=4294967295 uid=997 gid=995 euid=997 suid=997 fsuid=997 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295 comm="service" exe="/usr/bin/bash" subj=system_u:system_r:nrpe_t:s0 key=(null)
type=AVC msg=audit(1445982801.783:1446): avc:  denied  { getattr } for  pid=6117 comm="service" path="/usr/bin/systemctl" dev="dm-1" ino=2101040 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:systemd_systemctl_exec_t:s0 tclass=file
type=SYSCALL msg=audit(1445982801.783:1446): arch=c000003e syscall=4 success=no exit=-13 a0=1f17280 a1=7fffbdd7a1b0 a2=7fffbdd7a1b0 a3=2 items=0 ppid=6115 pid=6117 auid=4294967295 uid=997 gid=995 euid=997 suid=997 fsuid=997 egid=995 sgid=995 fsgid=995 tty=(none) ses=4294967295 comm="service" exe="/usr/bin/bash" subj=system_u:system_r:nrpe_t:s0 key=(null)

I thought this could be SELInux so I put it in permissive mode but still getting the same error.

[linuser]

As a test I bare-boned the check_init_script to something like this:

Code: Select all

#!/bin/sh
echo Hi
exit 0
 

And on the nagios server I can run the command with success.

Code: Select all

[root@labnagios ~]# /usr/lib64/nagios/plugins/check_nrpe -n -H 192.168.255.70 -c check_init1
Hi

Soooooo....seems its the output from the script that NRPE can't handle. I *think* its due to the way the older init scripts returned output. (you get a lot more with systemctl/systemd)

[linuser]

I changed the script again to a very basic one just to see how it looks in the nagios UI. I have added the file to this post. For some reason NRPE can read and process this. If I change up the script though to use systemctl with the full output NRPE can't read it. Here is what I did:

I changed this: (original)

Code: Select all

if [ $# -eq 1 ]; then
                        /sbin/service $1 status
                        ret=$?
                        case "$ret" in

to this: (modified)

Code: Select all

if [ $# -eq 1 ]; then
                        /bin/systemctl status $1 | awk 'NR==3'
                        ret=$?
                        case "$ret" in

The "awk 'NR==3' only gives me the 3rd line output, which is all I am interested in and it looks like this when you run the script from the remote host.

Code: Select all

[root@lab plugins]# /usr/lib64/nagios/plugins/check_init_service bgpd
   Active: active (running) since Tue 2015-10-27 10:40:44 CDT; 21h ago

One would think that NRPE would be able to process this like the barebones script but it can't. On the nagios server I get the same error as before.

Code: Select all

[root@labnagios plugins]# ./check_nrpe -n -H 192.168.255.70 -c check_init1
NRPE: Unable to read output

Here is the contents of the original full script for context.

Code: Select all

#!/bin/sh

PROGNAME=`basename $0`

print_usage() {
        echo "Usage: $PROGNAME"
}

print_help() {
        echo ""
        print_usage
        echo ""
        echo "This plugin checks the status of services normally started by the init process."
        echo ""
        support
        exit 0
}


case "$1" in
        --help)
                print_help
                exit 0
                ;;
        -h)
                print_help
                exit 0
                ;;
        *)

                if [ $# -eq 1 ]; then
                        /sbin/service $1 status
                        ret=$?
                        case "$ret" in
                             0)
                                exit $ret
                                ;;
                             *)
                                exit 2
                                ;;
                        esac
                else
                        echo "ERROR: No service name specified on command line"
                        exit 3
                fi
                ;;
esac

At this point I'd like to know a few things:

1) Why NRPE can process a basic "This is a test" script but it can't process the modified one that uses systmectl with a one line output?
2) What can I do to get this working? I can only see 2 options - either modify the system to use /sbin/service for these two services, or modify the script again to allow NRPE to properly read it.

Help is greatly appreciated.

[rkennedy]

Hm, this may be related to permissions. It would explain why your simple script worked as well. Can you try adding sudo before your command?

Code: Select all

command[check_init1]=sudo /usr/lib64/nagios/plugins/check_init_service bgpd
command[check_init2]=sudo /usr/lib64/nagios/plugins/check_init_service zebra