check_nrpe check_services returns "Nok"

[SavaSC]

I know you guys are getting tired of talking to me this week! But I'm trying to get this new Nagios server up and running smooth. Hopefully, I'll be done soon and you'll be able to help others.

I opened a previous thread on this issues (https://support.nagios.com/forum/viewto ... 16&t=44061) but it "autocorrected" itself and so the thread was closed. Well, it has spontaneously regenerated itself today. I did some more digging to see if I could fix it before bugging y'all again. The problem is that for a particular service, the check_nrpe check_services plugin(s) are telling me "Nok" (whatever that means). This only occurs when testing for this particular service using check_nrpe.

Here is the result of running the check_service command on the remote server to find both the init & indexserver services.

Capture.PNG

Here is the result of running check_service via check_nrpe from the Nagios server for both init & indexserver services.

Capture1.PNG

As you can see, the init service returns a "3" each time. However, the indexserver service returns a "1" on the remote server and a "Nok" on the Nagios server. I *assume* that it is the "1" that is throwing things off?

Any ideas of what I can check next?

[tgriep]

The "Nok" means that the check could not find that service running and returns a not OK status, the 1 means that only one indexserver service running.

It could be a permission problem as the NRPE agent runs the plugins as the nagios user.

To see if that is the issue, login to the hpfilridx01 server as root and run this command, post the output

Code: Select all

ps aax | egrep "indexserver" | grep -v egrep | wc -l | sed 's| ||g'

Then run it as the nagios user by running the following, post the output

Code: Select all

su nagios
ps aax | egrep "indexserver" | grep -v egrep | wc -l | sed 's| ||g'
/usr/local/nagios/libexec/check_services -p indexserver

Thanks.

[SavaSC]

Thank you for the explanation of Nok and what the numbers mean. I have attached a screenshot of the commands.

[tgriep]

It doesn't look like a permission problem.
Can you login to the remote server as root, run the following command and post the full output?

Code: Select all

ps -ef --cols=300

The simplest way to post the data is to highlight it in the window, copy it and paste it to this post in Code Wraps.
That will save time with the screen captures.

[SavaSC]

Here you go!

Code: Select all

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 May10 ?        00:00:09 init [3]
root         2     0  0 May10 ?        00:00:00 [kthreadd]
root         3     2  0 May10 ?        00:00:17 [ksoftirqd/0]
root         6     2  0 May10 ?        00:00:00 [migration/0]
root         7     2  0 May10 ?        00:00:02 [watchdog/0]
root         8     2  0 May10 ?        00:00:05 [migration/1]
root        10     2  0 May10 ?        00:00:10 [ksoftirqd/1]
root        12     2  0 May10 ?        00:00:01 [watchdog/1]
root        13     2  0 May10 ?        00:00:00 [migration/2]
root        15     2  0 May10 ?        00:00:14 [ksoftirqd/2]
root        16     2  0 May10 ?        00:00:02 [watchdog/2]
root        17     2  0 May10 ?        00:00:05 [migration/3]
root        19     2  0 May10 ?        00:00:12 [ksoftirqd/3]
root        20     2  0 May10 ?        00:00:01 [watchdog/3]
root        21     2  0 May10 ?        00:00:00 [cpuset]
root        22     2  0 May10 ?        00:00:00 [khelper]
root        23     2  0 May10 ?        00:00:00 [netns]
root        24     2  0 May10 ?        00:00:01 [sync_supers]
root        25     2  0 May10 ?        00:00:00 [bdi-default]
root        26     2  0 May10 ?        00:00:00 [kintegrityd]
root        27     2  0 May10 ?        00:00:00 [kblockd]
root        28     2  0 May10 ?        00:00:00 [md]
root        29     2  0 May10 ?        00:00:00 [khungtaskd]
root        30     2  0 May10 ?        00:00:00 [kswapd0]
root        31     2  0 May10 ?        00:00:00 [ksmd]
root        32     2  0 May10 ?        00:00:01 [khugepaged]
root        33     2  0 May10 ?        00:00:00 [fsnotify_mark]
root        34     2  0 May10 ?        00:00:00 [crypto]
root        38     2  0 May10 ?        00:00:00 [kthrotld]
root        40     2  0 May10 ?        00:00:00 [kpsmoused]
root        88     2  0 May10 ?        00:00:00 [khubd]
root       131     2  0 May10 ?        00:00:00 [kmpath_rdacd]
root       139     2  0 May10 ?        00:00:00 [kmpath_aluad]
root       229     2  0 May10 ?        00:00:00 [mpt_poll_0]
root       255     2  0 May10 ?        00:00:00 [mpt/0]
root       320     2  0 May10 ?        00:00:00 [scsi_eh_0]
root       321     2  0 May10 ?        00:00:00 [mpt_poll_1]
root       322     2  0 May10 ?        00:00:00 [mpt/1]
root       324     2  0 May10 ?        00:00:00 [scsi_eh_1]
root       325     2  0 May10 ?        00:00:00 [mpt_poll_2]
root       326     2  0 May10 ?        00:00:00 [mpt/2]
root       327     2  0 May10 ?        00:00:00 [scsi_eh_2]
root       421     2  0 May10 ?        00:00:06 [kjournald]
root       427     2  0 May10 ?        00:00:04 [flush-8:0]
root       521     1  0 May10 ?        00:00:00 /sbin/udevd --daemon
root       687     1  0 11:53 ?        00:00:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.init.pid
root       718     2  0 May10 ?        00:00:00 [ata_sff]
root       725     2  0 May10 ?        00:00:00 [vmmemctl]
root       763     2  0 May10 ?        00:00:12 [scsi_eh_3]
root       764     2  0 May10 ?        00:00:00 [scsi_eh_4]
root       836     2  0 May10 ?        00:00:00 [kauditd]
root       913   521  0 May10 ?        00:00:00 /sbin/udevd --daemon
root       915   521  0 May10 ?        00:00:00 /sbin/udevd --daemon
root       935     2  0 May10 ?        00:00:34 [kjournald]
root       936     2  0 May10 ?        00:00:14 [kjournald]
100       1361     1  0 May10 ?        00:00:02 /bin/dbus-daemon --system
root      1383     1  0 May10 ?        00:02:50 /sbin/syslog-ng
root      1386     1  0 May10 ?        00:00:08 /sbin/klogd -c 1 -x
106       1406     1  0 May10 ?        00:00:08 /usr/sbin/hald --daemon=yes
root      1409     1  0 May10 ?        00:00:00 /usr/sbin/console-kit-daemon
root      1410  1406  0 May10 ?        00:00:00 hald-runner
root      1492  1410  0 May10 ?        00:00:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event4
root      1525  1410  0 May10 ?        00:00:02 hald-addon-storage: no polling on /dev/fd0 because it is explicitly disabled
106       1530  1410  0 May10 ?        00:00:16 /usr/lib64/hal/hald-addon-acpi
root      1537  1410  0 May10 ?        00:00:23 hald-addon-storage: polling /dev/sr0 (every 16 sec)
root      1912     2  0 04:34 ?        00:00:01 [kworker/3:0]
root      2153     1  0 May10 ?        00:00:31 /sbin/haveged -w 1024 -v 1
root      2160     1  0 May10 ?        00:00:01 /sbin/rpcbind
root      2300     1  0 May10 ?        00:10:59 /usr/bin/vmtoolsd
root      2564     1  0 May10 ?        00:03:12 /usr/sbin/irqbalance
root      2578     1  0 May10 ?        01:08:51 /opt/novell/ganglia/monitor/sbin/gmond --conf=/etc/opt/novell/ganglia/monitor/gmond.conf
root      2582     1  0 May10 ?        00:00:00 startpar -f -- vmtoolsd
nobody    2585     1  0 May10 ?        01:38:12 /opt/novell/ganglia/monitor/sbin/gmetad --conf=/etc/opt/novell/ganglia/monitor/gmetad.conf
root      2596     2  0 May10 ?        00:05:43 [flush-8:16]
root      2597     2  0 May10 ?        00:00:09 [flush-8:32]
root      2738     1  0 May10 ?        00:00:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
ntp       2758     1  0 May10 ?        00:00:54 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -c /etc/ntp.conf
root      2761  2758  0 May10 ?        00:00:00 ntpd: asynchronous dns resolver
root      2774     1  0 May10 ?        00:00:09 /usr/sbin/nscd
root      2862     1  0 May10 ?        00:00:04 /usr/lib/postfix/master
postfix   2889  2862  0 May10 ?        00:00:00 qmgr -l -t fifo -u
root      2909     1  0 May10 ?        00:00:29 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun    2910  2909  0 May10 ?        00:00:01 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun    2911  2909  0 May10 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun    2912  2909  0 May10 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun    2913  2909  0 May10 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun    2914  2909  0 May10 ?        00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
root      2935     1  0 May10 ?        00:00:01 /usr/sbin/cron
104       3195     1  0 May10 ?        00:00:28 /usr/sbin/memcached -m 2048 -t 8 -d -l hpfilridx01.ltcsvc.com
root      3217     1  0 May10 ?        00:01:12 /usr/lib64/jvm/java/jre/bin/java -cp /opt/novell/datamodel-service/lib/* -Djava.library.path=/opt/novell/datamodel-service/lib -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=false -Djava.rmi.server.hostname=127.0.0.1 -Xdump:heap:label=/var/opt
108       3278     1  0 May10 ?        00:34:07 /usr/bin/java -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=false -Xdump:heap:label=/var/opt/novell/jetty/dumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:label=/var/opt/novell/jetty/dumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:sys
root      3282     1  0 May10 ?        00:00:00 startpar -f -- vabase-jetty9
root      3456     1  0 May10 tty1     00:00:00 /sbin/mingetty --noclear tty1
root      3457     1  0 May10 tty2     00:00:00 /sbin/mingetty tty2
root      3458     1  0 May10 tty3     00:00:00 /sbin/mingetty tty3
root      3459     1  0 May10 tty4     00:00:00 /sbin/mingetty tty4
root      3460     1  0 May10 tty5     00:00:00 /sbin/mingetty tty5
root      3461     1  0 May10 tty6     00:00:00 /sbin/mingetty tty6
root      5440     2  0 13:01 ?        00:00:00 [kworker/2:2]
postfix   5933  2862  0 13:09 ?        00:00:00 pickup -l -t fifo -u
root      7756 28643  0 13:37 pts/0    00:00:00 su nagios
nagios    7757  7756  0 13:37 pts/0    00:00:00 bash
root      9050     2  0 13:57 ?        00:00:00 [kworker/0:0]
root      9616     2  0 14:05 ?        00:00:00 [kworker/u:1]
root     10105     2  0 14:13 ?        00:00:00 [kworker/0:2]
root     10172     2  0 14:14 ?        00:00:00 [kworker/u:2]
root     10257     2  0 14:15 ?        00:00:00 [kworker/1:2]
root     10473     2  0 14:18 ?        00:00:00 [kworker/0:1]
root     10561     2  0 14:19 ?        00:00:00 [kworker/u:0]
root     10633     2  0 14:20 ?        00:00:00 [kworker/1:1]
root     10665     2  0 May29 ?        00:00:07 [kworker/3:2]
nagios   10763  7757  0 14:22 pts/0    00:00:00 ps -ef --cols=300
root     19158     2  0 08:37 ?        00:00:01 [kworker/1:0]
root     27230  2738  0 10:31 ?        00:00:00 sshd: root@notty
root     27236 27230  0 10:31 ?        00:00:00 /usr/lib64/ssh/sftp-server
root     27829  2738  0 10:39 ?        00:00:00 sshd: root@notty
root     27834 27829  0 10:39 ?        00:00:00 /usr/lib64/ssh/sftp-server
root     28633  2738  0 10:48 ?        00:00:00 sshd: root@pts/0
root     28643 28633  0 10:48 pts/0    00:00:00 -bash
root     28788     1  4 10:49 ?        00:09:31 /usr/lib64/jvm/java/bin/java -cp /opt/novell/search/indexserver/lib/novell-teaming-luceneserver.jar:/opt/novell/search/indexserver/lib/kablink-teaming-util.jar:/opt/novell/search/indexserver/lib/kablink-teaming-util-search.jar:/opt/novell/search/indexs
root     31202     2  0 May27 ?        00:00:08 [kworker/2:1]

[tgriep]

All of the tests look good and I can not recreate the error.
Can you run the check_nrpe command from the Nagios server and then see if the NRPE agent running on the remote server logs anyerrors to any of the files in the /var/log folder?

[SavaSC]

OK, I'm not sure what is going on, but the situation has fixed itself again. I'm not sure why, but the host just sometimes refuses to talk about that service. (Black sheep of the family???) When it decides to be obnoxious again, I'll have our resident Linux guru take a look at that server to see if he can see an issue.

[tgriep]

That is strange. If you find anything on what causes the issue, let us know.

[SavaSC]

Will do!

[tgriep]

Thanks