command not defined

[arenist]

Under xinetd

[arenist]

And restart of service xinetd doesn't matter. Nrpe's behavior is still the same.

[slansing]

If you are using xinetd you must make all of your configuration changes in "/etc/xinetd.d/nrpe" editing the /usr/local/nagios/etc/nrpe.cfg file will have no effect unless specified.

[arenist]

Hi slansing,

my /etc/xinetd.d/nrpe looks like this:

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg --inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1 194.59.103.56
}

There is a line server_args that tells nagios to use the configuration saved in /usr/local/nagios/etc/nrpe.cfg. The other checks that work fine via automatic calling like check_hda1 are not implemented here, too. 194.59.103.56 is the IP address of my nagios server.

Remember, my problem is that checks via CLI work while the automatic calls work do not. They only work with those checks that have been in /usr/local/nagios/etc/nrpe.cfg before I added some new checks.

***** Nagios *****

Notification Type: PROBLEM

Service: Check Listener
Host: VBGMADB11
Address: 194.59.101.179
State: CRITICAL

Date/Time: Tue Dec 10 03:08:23 CET 2013

Additional Info:

NRPE: Command check_lsnr not defined

BUT:
As user nagios from the nagios server:

Code: Select all

nagios@madpbk51:~> id
uid=200(nagios) gid=200(nagios) Gruppen=200(nagios),210(nagcmd),504(oinstall)
nagios@madpbk51:~> /usr/local/nagios/libexec/check_nrpe -H VBGMADB11 -c check_lsnr
Oracle Listener is working fine.
nagios@madpbk51:~> echo $?
0

As user nagios on the client server:

Code: Select all

[nagios@vbgmadb11 etc]$ id
uid=54322(nagios) gid=54323(nagios) groups=54321(oinstall),54323(nagios)
[nagios@vbgmadb11 etc]$ /usr/local/nagios/libexec/check_nrpe -H localhost -c check_lsnr
Oracle Listener is working fine.
[nagios@vbgmadb11 etc]$ echo $?
0
[nagios@vbgmadb11 etc]$ /usr/local/nagios/libexec/check_listener.bsh
Oracle Listener is working fine.
[nagios@vbgmadb11 etc]$ echo $?
0

So I think the configuration of nagios and nrpe might be OK. And if it's OK, nagios does not look at the "real" /usr/local/nagios/etc/nrpe.cfg. Instead of the file nagios uses some known configuration which is saved only God knows where, I assume.

Yesterday I tried a new nagios installation of the server and a new nrpe installation on the client. But the situation is still the same.

Regards Arenist

[abrist]

Lets check to see if you have more than one nrpe installation and config file:

Code: Select all

yum install mlocate
updatedb
locate nrpe.cfg

And lets check for more than one running nrpe daemon:

Code: Select all

ps -aef | grep "nrpe\|xinetd"

[lmiltchev]

Run:

Code: Select all

service xinetd stop
killall nrpe

Verify nrpe is not running:

Code: Select all

ps axuw | grep nrpe

Start the daemon:

Code: Select all

service xinetd start

Let us know if this fixed your problem.

[arenist]

Hi abrist,

it did not work. Here's what I did:

Client server

Code: Select all

[root@vbgmadb11 local]# yum install mlocate
Loaded plugins: security
Setting up Install Process
Package mlocate-0.15-1.el5.2.x86_64 already installed and latest version
Nothing to do
[root@vbgmadb11 local]# updatedb &
[1] 544
[root@vbgmadb11 local]#
[1]+  Done                    updatedb
[root@vbgmadb11 local]# locate nrpe.cfg
/usr/local/nagios/etc/nrpe.cfg

Code: Select all

[root@vbgmadb11 local]# ps -aef | grep "nrpe\|xinetd"
root       685   427  0 07:21 pts/0    00:00:00 grep nrpe\|xinetd
root     31835     1  0 Dec09 ?        00:00:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
[root@vbgmadb11 local]# service xinetd stop
Stopping xinetd:                                           [  OK  ]
[root@vbgmadb11 local]# killall nrpe
nrpe: no process killed

Code: Select all

[root@vbgmadb11 local]# ps axuw | grep nrpe
root       825  0.0  0.0  61180   756 pts/0    R+   07:22   0:00 grep nrpe

Code: Select all

[root@vbgmadb11 local]# service xinetd start
Starting xinetd:                                           [  OK  ]

On the nagios server I want the client vbgmadb11 to be treated again by nagios

Code: Select all

[root@vbgmadb11 local]# madpbk51:~ # vi /usr/local/nagios/etc/servers/services.cfg
...
define service{
        use generic-service
        hostgroup_name                  DB
        host_name                       !vmacldb92,!vbgmadb71;,!vbgmadb11
        service_description             Check Listener
        contact_groups                  admins,db-admins
        normal_check_interval           60
        notification_interval           240
        check_command                   check_nrpe!check_lsnr
}
...

madpbk51:~ # /etc/init.d/nagios stop
Stopping nagios: .done.
madpbk51:~ # /etc/init.d/nagios start
Starting nagios: done.

Here are some logs:
syslog of the server:

Code: Select all

Dec 11 07:57:26 madpbk51 nagios: Caught SIGTERM, shutting down...
Dec 11 07:57:26 madpbk51 nagios: Successfully shutdown... (PID=32177)
Dec 11 07:57:58 madpbk51 su: (to nagios) root on /dev/pts/0
Dec 11 07:57:58 madpbk51 nagios: Nagios 3.5.0 starting... (PID=3164)
Dec 11 07:57:58 madpbk51 nagios: Local time is Wed Dec 11 07:57:58 CET 2013
Dec 11 07:57:58 madpbk51 nagios: LOG VERSION: 2.0
Dec 11 07:57:58 madpbk51 nagios: Finished daemonizing... (New PID=3165)

nagios log of the server:

Code: Select all

/usr/local/nagios/var # tail -f nagios.log
[1386743263] Caught SIGTERM, shutting down...
[1386743263] Successfully shutdown... (PID=2747)
[1386743276] Nagios 3.5.0 starting... (PID=32176)
[1386743276] Local time is Wed Dec 11 07:27:56 CET 2013
[1386743276] LOG VERSION: 2.0
[1386743276] Finished daemonizing... (New PID=32177)
...
[1386744486] SERVICE ALERT: vbgmadb11;Check Listener;CRITICAL;SOFT;1;NRPE: Command 'check_lsnr' not defined
...
[1386744606] SERVICE ALERT: vbgmadb11;Check Listener;CRITICAL;SOFT;2;NRPE: Command 'check_lsnr' not defined
[1386744726] SERVICE ALERT: vbgmadb11;Check Listener;CRITICAL;HARD;3;NRPE: Command 'check_lsnr' not defined
...
[1386744726] SERVICE NOTIFICATION: nagiosadmin;vbgmadb11;Check Listener;CRITICAL;notify-service-by-email;NRPE: Command check_lsnr not defined

syslog of the client:

Code: Select all

Dec 11 07:22:04 vbgmadb11 xinetd[31835]: Exiting...
Dec 11 07:23:21 vbgmadb11 xinetd[947]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Dec 11 07:23:21 vbgmadb11 xinetd[947]: Started working: 1 available service
Dec 11 07:55:31 vbgmadb11 xinetd[947]: START: nrpe pid=4372 from=194.59.103.56
Dec 11 07:55:31 vbgmadb11 nrpe[4372]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Dec 11 07:55:31 vbgmadb11 xinetd[947]: EXIT: nrpe status=0 pid=4372 duration=0(sec)

I logged in as user nagios on the server and tried

Code: Select all

nagios@madpbk51:~> /usr/local/nagios/libexec/check_nrpe -H vbgmadb11 -c check_lsnr
Oracle Listener is working fine.

Maybe the logging can help you. I really don't know what's wrong. 14 checks run correctly, any new added does not.

Regards Arenist

[abrist]

Was the locate run on the remote host? If not, it should have been and you should rerun it on the remote host, not vbgmadb11 .
Something is wrong with the nrpe.cfg configuration on the remote host. Could you please post it?

[arenist]

Hi abrist,

vbgmadb11 is the remote host (client) and madpbk51 is the nagios server.

I did already post the nrpe.cgf of vbgmadb11 (on page 1 at Fri Dec 06, 2013 2:21 pm)

If there's something wrong with my nrpe.cfg, why doesn't a check via CLI fail, too?

Regards arenist

PS: I've found some interesting information. When looking at the nagions server's var directory I noticed that after activating the service check_lsnr for vbgmadb11 and reload of nagios the service was found in /usr/local/nagios/var/objects.cache, but not in retention.dat. Might this be a hint?

Both files were modified at 10:07 am when I reloaded nagios.

Code: Select all

madpbk51:/usr/local/nagios/var # ll
total 1772
drwxrwxr-x 2 nagios nagios   4096 Dec 12 00:00 archives
-rw-r--r-- 1 nagios nagios    976 Dec 12 10:07 nagios.debug
-rw-r--r-- 1 nagios nagios      6 Dec 12 10:07 nagios.lock
-rw-r--r-- 1 nagios nagios 100416 Dec 12 10:07 nagios.log
-rw-r--r-- 1 nagios nagios 399468 Dec 12 10:07 objects.cache
-rw------- 1 nagios nagios 635632 Dec 12 10:07 retention.dat
drwxrwsr-x 2 nagios nagcmd   4096 Dec 12 10:07 rw
drwxrwxr-x 3 nagios nagios   4096 Dec  9 13:57 spool

[abrist]

I am not sure if the retention.dat conjecture is relevant. If it works from the nagios server cli, if must be a configuration issue in the nagios configs. I have looked through the configs on teh previous page and I don't see any discrepancy. When is the last time you restarted the nagios process?