Passing an argument containing a minus sign

[sreinhardt]

To clarify what you might be seeing, nrpe forks on each new connection, and without investigating further it may like core fork again for the actual check it runs. Thus if nagios were already checking one item, you are running a manual nrpe check, and the daemon is running, it would make sense that you see 3 nrpe processes. At the same time, if it does fork on each check, just the daemon, the fork for your connection, and the fork for check_procs to be run would make the three processes. At the very least I would expect that when using check_nrpe to check the number of nrpe processes you should always see 2 processes, if not 3. As you might expect, this number will constantly fluctuate depending on the number of actively running checks nrpe is running and could certainly be adding to the confusion.

[joedebattista]

Here's the results of the ps on the nagios server.
[joed@vx8 ~]$ hostname
vx8
[joed@vx8 ~]$ ps -aef | grep nrpe
nagios 24281 24280 0 12:03 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.141.12 -c check_local_disk -a 10% 5% /var
nagios 24290 24289 0 12:03 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 169.237.234.14 -c check_local_disk -a 10% 5% /var
nagios 24296 24295 0 12:03 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 169.237.234.13 -c check_local_procs_byname -a ovjboss
nagios 24299 24298 0 12:03 ? 00:00:00 sh -c usr/local/nagios/libexec/check_nrpe -H 169.237.234.15 -c check_aixpaths -a 95 98
nagios 24302 24301 0 12:03 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.136.8 -c check_local_disk -a 10% 5% /home
joed 24304 21169 0 12:03 pts/1 00:00:00 grep nrpe

I want to monitor the amount of nrpe daemons that are running on the remote host. Here's the ps from that host.
$ hostname
i22m6
$ ps -aef | grep nrpe
joed 7995462 12648518 1 12:03:19 pts/1 0:00 grep nrpe
nagios 36831366 1 0 16:16:24 - 0:03 /usr/local/nagios/nrpe -c /usr/local/nagios/nrpe.cfg -d

[abrist]

Well, the nrpe daemon is one, the remote connection is two, and then the ps is three. Seem right?

[joedebattista]

Hmm.. It looks like my posting from yesterday didn't get here.

Here's the results of my ps on the nagios server:
[joed@vx8 ~]$ ps -aef | grep nrpe
nagios 28205 28204 0 12:00 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.134.47 -c check_disk -a c: 4% 2%
nagios 28208 28207 0 12:00 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.132.20 -c check_disk -a c: 10% 5%
nagios 28211 28210 0 12:00 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.142.110 -c check_win_all_services -a /v
nagios 28214 28213 0 12:00 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.142.148 -c check_disk -a c: 10% 5%
nagios 28217 28216 0 12:00 ? 00:00:00 usr/local/nagios/libexec/check_nrpe -H 64.54.142.160 -c check_win_all_services -a /v
joed 28219 20350 0 12:00 pts/1 00:00:00 grep nrpe


Here's the ps on the remote host (i22m6), which is where I'm checking to see the number of nrpe daemons. There should only be one there.
$ ps -aef | grep nrpe
joed 18153502 22806572 1 12:00:29 pts/1 0:00 grep nrpe
nagios 31981800 1 0 13:51:57 - 0:04 /usr/local/nagios/nrpe -c /usr/local/nagios/nrpe.cfg -d

Again, here's the results when I run my command on the remote host:
i22m6:/usr/local/nagios/libexec/check_procs -c 1:1 -C nrpe -a nrpe.cfg
PROCS OK: 1 process with command name 'nrpe', args 'nrpe.cfg'

and here's the command I issue from the nagios server:
vx8:/usr/local/nagios/libexec/check_nrpe -H i22m6 -c check_4nrped2 -a "1:1" "nrpe" "nrpe.cfg"
PROCS CRITICAL: 3 processes with command name 'nrpe', args 'nrpe.cfg'

This sucker is driving me bananas!

[abrist]

The reason for the multiple processes, as I and Spenser have pointed out earlier in this thread, is due to how you are checking for nrpe - specifically, you are checking nrpe *through* nrpe with a check command run *by* nrpe. This will cause two additional nrpe processes to spawn in addition to the daemon:
1. The nrpe daemon
2. The remote nrpe connection from the nagios server
3. The command run by the nrpe check.
All three of these will meet your process string check, but you will only see the latter two when running a check *through* nrpe.
Also, this check will have other problems if more than one nrpe check is run on the host at the same time it will cause the process count to rise further. What is the goal of this check?
You will not see these extra processes when running a ps *unless* the nagios server is doing an nrpe check *at that exact moment*.

[eloyd]

And I think this was pointed out as well, if you're checking for NRPE and Nagios runs an NRPE check, you will see more than one NRPE process. More like 4. I think you need to ask what the purpose of your check is. It sounds like you want to confirm that NRPE is up and running.

This would be more easily obtained by simply running check_nrpe -H hostname and seeing if you get an OK or a CRITICAL. If you want to go farther, you can check_tcp -H hostname -p 5666 to see that it is responding on the correct port. If you want to add to it even more, you can use check_procs to see that there is at least one process running. But checking to see that exactly one process is running is sort of like checking to make sure that there is exactly one HTTP process running. Potentially every new request could very well make a new process, all of which were spawned from a particular parent.

[abrist]

Apt, on point eloyd. Checking for number of instances of a process is usually rather pointless when dealing with processes that fork themselves. You may check for extremely high/low quantities, but anything else in between could very well vary at any given moment.

[joedebattista]

Thanks for the information. The reason I was trying to monitor the number of nrpe processes was that, occasionally, the nrpe processes on some of my AIX hosts don't seem to complete. When I do a "top" or "topas" the processes are taking up the majority of the CPU. Killing the multiple nrpe processes and then restarting nrpe seems to fix the problem. Unfortunately, my users usually discover this before I do.

[eloyd]

Then you need to use check_procs.

Look at /usr/local/nagios/libexec/check_procs -h and look at this part:

Code: Select all

 -m, --metric=TYPE
  Check thresholds against metric. Valid types:
  PROCS   - number of processes (default)
  VSZ     - virtual memory size
  RSS     - resident set memory size
  CPU     - percentage cpu
  ELAPSED - time elapsed in seconds

Specifically in one of the examples:

Code: Select all

 check_procs -w 10 -c 20 --metric=CPU
  Alert if cpu of any processes over 10%% or 20%%

[tmcdonald]

joe, let us know if check_procs clears things up for you.