Page 1 of 1
check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 12:36 pm
by SavaSC
I know you guys are getting tired of talking to me this week! But I'm trying to get this new Nagios server up and running smooth. Hopefully, I'll be done soon and you'll be able to help others.
I opened a previous thread on this issues (
https://support.nagios.com/forum/viewto ... 16&t=44061) but it "autocorrected" itself and so the thread was closed. Well, it has spontaneously regenerated itself today. I did some more digging to see if I could fix it before bugging y'all again. The problem is that for a particular service, the check_nrpe check_services plugin(s) are telling me "Nok" (whatever that means). This only occurs when testing for this particular service using check_nrpe.
Here is the result of running the check_service command on the remote server to find both the init & indexserver services.
Capture.PNG
Here is the result of running check_service via check_nrpe from the Nagios server for both init & indexserver services.
Capture1.PNG
As you can see, the init service returns a "3" each time. However, the indexserver service returns a "1" on the remote server and a "Nok" on the Nagios server. I *assume* that it is the "1" that is throwing things off?
Any ideas of what I can check next?
Re: check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 1:14 pm
by tgriep
The "Nok" means that the check could not find that service running and returns a not OK status, the 1 means that only one indexserver service running.
It could be a permission problem as the NRPE agent runs the plugins as the nagios user.
To see if that is the issue, login to the hpfilridx01 server as root and run this command, post the output
Code: Select all
ps aax | egrep "indexserver" | grep -v egrep | wc -l | sed 's| ||g'
Then run it as the nagios user by running the following, post the output
Code: Select all
su nagios
ps aax | egrep "indexserver" | grep -v egrep | wc -l | sed 's| ||g'
/usr/local/nagios/libexec/check_services -p indexserver
Thanks.
Re: check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 1:40 pm
by SavaSC
Thank you for the explanation of Nok and what the numbers mean. I have attached a screenshot of the commands.
Re: check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 2:04 pm
by tgriep
It doesn't look like a permission problem.
Can you login to the remote server as root, run the following command and post the full output?
The simplest way to post the data is to highlight it in the window, copy it and paste it to this post in Code Wraps.
That will save time with the screen captures.
Re: check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 2:23 pm
by SavaSC
Here you go!
Code: Select all
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 May10 ? 00:00:09 init [3]
root 2 0 0 May10 ? 00:00:00 [kthreadd]
root 3 2 0 May10 ? 00:00:17 [ksoftirqd/0]
root 6 2 0 May10 ? 00:00:00 [migration/0]
root 7 2 0 May10 ? 00:00:02 [watchdog/0]
root 8 2 0 May10 ? 00:00:05 [migration/1]
root 10 2 0 May10 ? 00:00:10 [ksoftirqd/1]
root 12 2 0 May10 ? 00:00:01 [watchdog/1]
root 13 2 0 May10 ? 00:00:00 [migration/2]
root 15 2 0 May10 ? 00:00:14 [ksoftirqd/2]
root 16 2 0 May10 ? 00:00:02 [watchdog/2]
root 17 2 0 May10 ? 00:00:05 [migration/3]
root 19 2 0 May10 ? 00:00:12 [ksoftirqd/3]
root 20 2 0 May10 ? 00:00:01 [watchdog/3]
root 21 2 0 May10 ? 00:00:00 [cpuset]
root 22 2 0 May10 ? 00:00:00 [khelper]
root 23 2 0 May10 ? 00:00:00 [netns]
root 24 2 0 May10 ? 00:00:01 [sync_supers]
root 25 2 0 May10 ? 00:00:00 [bdi-default]
root 26 2 0 May10 ? 00:00:00 [kintegrityd]
root 27 2 0 May10 ? 00:00:00 [kblockd]
root 28 2 0 May10 ? 00:00:00 [md]
root 29 2 0 May10 ? 00:00:00 [khungtaskd]
root 30 2 0 May10 ? 00:00:00 [kswapd0]
root 31 2 0 May10 ? 00:00:00 [ksmd]
root 32 2 0 May10 ? 00:00:01 [khugepaged]
root 33 2 0 May10 ? 00:00:00 [fsnotify_mark]
root 34 2 0 May10 ? 00:00:00 [crypto]
root 38 2 0 May10 ? 00:00:00 [kthrotld]
root 40 2 0 May10 ? 00:00:00 [kpsmoused]
root 88 2 0 May10 ? 00:00:00 [khubd]
root 131 2 0 May10 ? 00:00:00 [kmpath_rdacd]
root 139 2 0 May10 ? 00:00:00 [kmpath_aluad]
root 229 2 0 May10 ? 00:00:00 [mpt_poll_0]
root 255 2 0 May10 ? 00:00:00 [mpt/0]
root 320 2 0 May10 ? 00:00:00 [scsi_eh_0]
root 321 2 0 May10 ? 00:00:00 [mpt_poll_1]
root 322 2 0 May10 ? 00:00:00 [mpt/1]
root 324 2 0 May10 ? 00:00:00 [scsi_eh_1]
root 325 2 0 May10 ? 00:00:00 [mpt_poll_2]
root 326 2 0 May10 ? 00:00:00 [mpt/2]
root 327 2 0 May10 ? 00:00:00 [scsi_eh_2]
root 421 2 0 May10 ? 00:00:06 [kjournald]
root 427 2 0 May10 ? 00:00:04 [flush-8:0]
root 521 1 0 May10 ? 00:00:00 /sbin/udevd --daemon
root 687 1 0 11:53 ? 00:00:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.init.pid
root 718 2 0 May10 ? 00:00:00 [ata_sff]
root 725 2 0 May10 ? 00:00:00 [vmmemctl]
root 763 2 0 May10 ? 00:00:12 [scsi_eh_3]
root 764 2 0 May10 ? 00:00:00 [scsi_eh_4]
root 836 2 0 May10 ? 00:00:00 [kauditd]
root 913 521 0 May10 ? 00:00:00 /sbin/udevd --daemon
root 915 521 0 May10 ? 00:00:00 /sbin/udevd --daemon
root 935 2 0 May10 ? 00:00:34 [kjournald]
root 936 2 0 May10 ? 00:00:14 [kjournald]
100 1361 1 0 May10 ? 00:00:02 /bin/dbus-daemon --system
root 1383 1 0 May10 ? 00:02:50 /sbin/syslog-ng
root 1386 1 0 May10 ? 00:00:08 /sbin/klogd -c 1 -x
106 1406 1 0 May10 ? 00:00:08 /usr/sbin/hald --daemon=yes
root 1409 1 0 May10 ? 00:00:00 /usr/sbin/console-kit-daemon
root 1410 1406 0 May10 ? 00:00:00 hald-runner
root 1492 1410 0 May10 ? 00:00:00 hald-addon-input: Listening on /dev/input/event0 /dev/input/event4
root 1525 1410 0 May10 ? 00:00:02 hald-addon-storage: no polling on /dev/fd0 because it is explicitly disabled
106 1530 1410 0 May10 ? 00:00:16 /usr/lib64/hal/hald-addon-acpi
root 1537 1410 0 May10 ? 00:00:23 hald-addon-storage: polling /dev/sr0 (every 16 sec)
root 1912 2 0 04:34 ? 00:00:01 [kworker/3:0]
root 2153 1 0 May10 ? 00:00:31 /sbin/haveged -w 1024 -v 1
root 2160 1 0 May10 ? 00:00:01 /sbin/rpcbind
root 2300 1 0 May10 ? 00:10:59 /usr/bin/vmtoolsd
root 2564 1 0 May10 ? 00:03:12 /usr/sbin/irqbalance
root 2578 1 0 May10 ? 01:08:51 /opt/novell/ganglia/monitor/sbin/gmond --conf=/etc/opt/novell/ganglia/monitor/gmond.conf
root 2582 1 0 May10 ? 00:00:00 startpar -f -- vmtoolsd
nobody 2585 1 0 May10 ? 01:38:12 /opt/novell/ganglia/monitor/sbin/gmetad --conf=/etc/opt/novell/ganglia/monitor/gmetad.conf
root 2596 2 0 May10 ? 00:05:43 [flush-8:16]
root 2597 2 0 May10 ? 00:00:09 [flush-8:32]
root 2738 1 0 May10 ? 00:00:00 /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
ntp 2758 1 0 May10 ? 00:00:54 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -c /etc/ntp.conf
root 2761 2758 0 May10 ? 00:00:00 ntpd: asynchronous dns resolver
root 2774 1 0 May10 ? 00:00:09 /usr/sbin/nscd
root 2862 1 0 May10 ? 00:00:04 /usr/lib/postfix/master
postfix 2889 2862 0 May10 ? 00:00:00 qmgr -l -t fifo -u
root 2909 1 0 May10 ? 00:00:29 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 2910 2909 0 May10 ? 00:00:01 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 2911 2909 0 May10 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 2912 2909 0 May10 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 2913 2909 0 May10 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
wwwrun 2914 2909 0 May10 ? 00:00:00 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf
root 2935 1 0 May10 ? 00:00:01 /usr/sbin/cron
104 3195 1 0 May10 ? 00:00:28 /usr/sbin/memcached -m 2048 -t 8 -d -l hpfilridx01.ltcsvc.com
root 3217 1 0 May10 ? 00:01:12 /usr/lib64/jvm/java/jre/bin/java -cp /opt/novell/datamodel-service/lib/* -Djava.library.path=/opt/novell/datamodel-service/lib -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=false -Djava.rmi.server.hostname=127.0.0.1 -Xdump:heap:label=/var/opt
108 3278 1 0 May10 ? 00:34:07 /usr/bin/java -Dfile.encoding=UTF-8 -Djava.rmi.server.useCodebaseOnly=false -Xdump:heap:label=/var/opt/novell/jetty/dumps/heapdump.%Y%m%d.%H%M%S.%pid.%seq.phd -Xdump:java:label=/var/opt/novell/jetty/dumps/javacore.%Y%m%d.%H%M%S.%pid.%seq.txt -Xdump:sys
root 3282 1 0 May10 ? 00:00:00 startpar -f -- vabase-jetty9
root 3456 1 0 May10 tty1 00:00:00 /sbin/mingetty --noclear tty1
root 3457 1 0 May10 tty2 00:00:00 /sbin/mingetty tty2
root 3458 1 0 May10 tty3 00:00:00 /sbin/mingetty tty3
root 3459 1 0 May10 tty4 00:00:00 /sbin/mingetty tty4
root 3460 1 0 May10 tty5 00:00:00 /sbin/mingetty tty5
root 3461 1 0 May10 tty6 00:00:00 /sbin/mingetty tty6
root 5440 2 0 13:01 ? 00:00:00 [kworker/2:2]
postfix 5933 2862 0 13:09 ? 00:00:00 pickup -l -t fifo -u
root 7756 28643 0 13:37 pts/0 00:00:00 su nagios
nagios 7757 7756 0 13:37 pts/0 00:00:00 bash
root 9050 2 0 13:57 ? 00:00:00 [kworker/0:0]
root 9616 2 0 14:05 ? 00:00:00 [kworker/u:1]
root 10105 2 0 14:13 ? 00:00:00 [kworker/0:2]
root 10172 2 0 14:14 ? 00:00:00 [kworker/u:2]
root 10257 2 0 14:15 ? 00:00:00 [kworker/1:2]
root 10473 2 0 14:18 ? 00:00:00 [kworker/0:1]
root 10561 2 0 14:19 ? 00:00:00 [kworker/u:0]
root 10633 2 0 14:20 ? 00:00:00 [kworker/1:1]
root 10665 2 0 May29 ? 00:00:07 [kworker/3:2]
nagios 10763 7757 0 14:22 pts/0 00:00:00 ps -ef --cols=300
root 19158 2 0 08:37 ? 00:00:01 [kworker/1:0]
root 27230 2738 0 10:31 ? 00:00:00 sshd: root@notty
root 27236 27230 0 10:31 ? 00:00:00 /usr/lib64/ssh/sftp-server
root 27829 2738 0 10:39 ? 00:00:00 sshd: root@notty
root 27834 27829 0 10:39 ? 00:00:00 /usr/lib64/ssh/sftp-server
root 28633 2738 0 10:48 ? 00:00:00 sshd: root@pts/0
root 28643 28633 0 10:48 pts/0 00:00:00 -bash
root 28788 1 4 10:49 ? 00:09:31 /usr/lib64/jvm/java/bin/java -cp /opt/novell/search/indexserver/lib/novell-teaming-luceneserver.jar:/opt/novell/search/indexserver/lib/kablink-teaming-util.jar:/opt/novell/search/indexserver/lib/kablink-teaming-util-search.jar:/opt/novell/search/indexs
root 31202 2 0 May27 ? 00:00:08 [kworker/2:1]
Re: check_nrpe check_services returns "Nok"
Posted: Wed May 31, 2017 3:12 pm
by tgriep
All of the tests look good and I can not recreate the error.
Can you run the check_nrpe command from the Nagios server and then see if the NRPE agent running on the remote server logs anyerrors to any of the files in the /var/log folder?
Re: check_nrpe check_services returns "Nok"
Posted: Tue Jun 06, 2017 12:31 pm
by SavaSC
OK, I'm not sure what is going on, but the situation has fixed itself again. I'm not sure why, but the host just sometimes refuses to talk about that service. (Black sheep of the family???) When it decides to be obnoxious again, I'll have our resident Linux guru take a look at that server to see if he can see an issue.
Re: check_nrpe check_services returns "Nok"
Posted: Tue Jun 06, 2017 12:40 pm
by tgriep
That is strange. If you find anything on what causes the issue, let us know.
Re: check_nrpe check_services returns "Nok"
Posted: Tue Jun 06, 2017 12:43 pm
by SavaSC
Will do!
Re: check_nrpe check_services returns "Nok"
Posted: Tue Jun 06, 2017 1:22 pm
by tgriep
Thanks