What am I missing? (NRPE: Unable to read output)

Support forum for Nagios Core, Nagios Plugins, NCPA, NRPE, NSCA, NDOUtils and more. Engage with the community of users including those using the open source solutions.
User avatar
tgriep
Madmin
Posts: 9177
Joined: Thu Oct 30, 2014 9:02 am

Re: What am I missing? (NRPE: Unable to read output)

Post by tgriep »

I think the problem is with the ! in the output of the plugin. The ! is used as a delimiter in Nagios and that could cause the error.
Try replacing the check_mem on your system with the one in the link below and see if that works for you.
https://raw.githubusercontent.com/justi ... eck_mem.pl
Be sure to check out our Knowledgebase for helpful articles and solutions!
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

lmiltchev wrote:What commands did you run in order to install Nagios Plugins and NRPE?

It is very strange that the nagios user is not available...
This account is currently not available.
What is the output of the following commands?

Code: Select all

grep nag /etc/passwd
grep nag /etc/group
grep "nrpe_" /path/to/the/nrpe.cfg
chage nagios -l
Directions to install were from here: http://tecadmin.net/install-nrpe-on-centos-rhel/
Same directions were used on each server.

I found it odd too that nagios wasn't available, but that's also consistent with the other servers.
grep nag /etc/passwd:

Code: Select all

nagios:x:498:499::/var/spool/nagios:/sbin/nologin
grep nag /etc/group:

Code: Select all

nagios:x:499:
grep "nrpe_" /etc/nagios/nrpe.cfg:

Code: Select all

nrpe_user=nrpe
nrpe_group=nrpe
chage nagios -l

Code: Select all

Last password change                                    : Mar 08, 2016
Password expires                                        : never
Password inactive                                       : never
Account expires                                         : never
Minimum number of days between password change          : -1
Maximum number of days between password change          : -1
Number of days of warning before password expires       : -1
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

tgriep wrote:I think the problem is with the ! in the output of the plugin. The ! is used as a delimiter in Nagios and that could cause the error.
Try replacing the check_mem on your system with the one in the link below and see if that works for you.
https://raw.githubusercontent.com/justi ... eck_mem.pl
I had that one stored to use, just in case (though I think I got it from somewhere else). But anyway, I installed the check_mem.pl plugin. For grins, I also copied over the check_mem plugin from another server that is working fine. Both have the same owners/groups/privs as the rest of the plugins. Still same error after restarting NRPE.
Updated NRPE.cfg as such:

Code: Select all

#command[check_mem]=/usr/lib64/nagios/plugins/check_mem  -f -w 20 -c 10
command[check_mem]=/usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10
Performed the restart of the NRPE service, still nothing.

If I run the /usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10 from the client, I get the following:

Code: Select all

<b>CRITICAL: Memory Usage (W> 20, C> 10): 99% <br>Swap Usage (W> 100, C> 100): 0%</b>|MemUsed=99%;20;10 SwapUsed=0%;100;100
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: What am I missing? (NRPE: Unable to read output)

Post by hsmith »

aneah wrote: If I run the /usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10 from the client, I get the following:
What user did you run this as?
Former Nagios Employee.
me.
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

hsmith wrote:
aneah wrote: If I run the /usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10 from the client, I get the following:
What user did you run this as?
root.
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

hsmith wrote:
aneah wrote: If I run the /usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10 from the client, I get the following:
What user did you run this as?
I also managed to find a way to su the Nagios account. For future reference for others, use the following command:

Code: Select all

su - nagios -s /bin/bash
I also ran the /usr/lib64/nagios/plugins/check_mem.pl -w 20 -c 10 as Nagios, from the client and obtained the same results as previously.

Code: Select all

<b>CRITICAL: Memory Usage (W> 20, C> 10): 99% <br>Swap Usage (W> 100, C> 100): 0%</b>|MemUsed=99%;20;10 SwapUsed=0%;100;100
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: What am I missing? (NRPE: Unable to read output)

Post by hsmith »

It looks like you have NRPE configured to run as the NRPE user instead of the nagios user, can you su to that user and try the check?
Former Nagios Employee.
me.
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

hsmith wrote:It looks like you have NRPE configured to run as the NRPE user instead of the nagios user, can you su to that user and try the check?
Exact same results as running as Nagios and root. In fact, the NRPE account gave the same error as the Nagios account and had to be run as I previously indicated.

Code: Select all

<b>CRITICAL: Memory Usage (W> 20, C> 10): 99% <br>Swap Usage (W> 100, C> 100): 0%</b>|MemUsed=99%;20;10 SwapUsed=0%;100;100
Also, NRPE is the same user in the NRPE.cfg file on all the other servers. Again, this is ONLY the check_mem plugin on one server. Others running on the same server all have the same owner, group, and permissions.

In fact, I just copied over the check_mem plugin and NRPE.cfg files from another server (where things were functioning fine) and restarted the nrpe process and no change.

I'll be doing an update and reboot on the machine tonight. I'm hoping that automagically fixes this.
User avatar
hsmith
Agent Smith
Posts: 3539
Joined: Thu Jul 30, 2015 11:09 am
Location: 127.0.0.1
Contact:

Re: What am I missing? (NRPE: Unable to read output)

Post by hsmith »

Here's hoping.

Something to do would be compare the output of a cat /proc/meminfo command on both your working servers and a non working server to see if there is some weird formatting the plugin could be tripping on.
Former Nagios Employee.
me.
User avatar
aneah
Posts: 12
Joined: Mon Apr 04, 2016 8:55 am
Location: Norman, OK

Re: What am I missing? (NRPE: Unable to read output)

Post by aneah »

Officially resolved as of tonight.
Cause: At this point, just speculation.
Remedy: Updated RHEL from 6.6 to 6.7 on the client and rebooted. When the server came back up, the error was gone.
Locked